awk on S Anand

How to review trending GitHub repos on VS Code

Sat, 27 Sep 2025 13:17:50 +0000

Here’s how I track trending GitHub repos each week. I run a scheduled script that saves a clean TSV I can scan fast.

It uses uvx gtrending to fetch weekly trending repos for:

Rust: High-quality system tools. (Anything in Rust seems cool.)
Go: Reliable CLI/infra tools. (Like Rust, most Go code seems good.)
Python: Most AI/ML stuff
TypeScript: Most modern JS codebases
JavaScript: Most front-end utilities
Shell: Productivity scripts

I pipe results through jq to extract:

Language
Stars: for popularity
Current period stars: for growth
Date: when I run it
Full name
Description

I use awk to de-duplicate by repo and save into a TSV file.

Then I review the TSV in VS Code. I stopped using Excel. Rainbow CSV and Edit CSV make large TSVs easy to scan, sort, and re-structure. (I’m quite excited at the new tricks I’m learning to replace Excel!)

I tag the first column with:

🟣 pending
🔴 ignore (with reason in the last column)
🟢 active (I currently use it)
⏺️ review

I picked these icons carefully. A descending sort puts 🟣 on top (to scan quickly) and ⏺️ at the bottom (for closer review). This also puts top-starred repos in each language first.

The payoff for the right tools is high. This process makes it easy. I found Flameshot, duf and wrkflw this way.

To try it, download and run bash trending-repos.sh and open the TSV it generates.

Faster data crunching

Fri, 23 Sep 2011 18:20:10 +0000

I’ve been playing with big data lately.

The good part is, it’s easy to get interesting results. The data is so unwieldy that even average value calculations provoke a “Amazing! I didn’t know that,” response (No exaggeration. I heard this from two separate ~ $1bn businesses this month.)

The bad part is that calculating even that simple average is slow.

For example, take this 40MB file (380MB unzipped) and extract the first column.

The simplest Python script to get the first column looks like this:

```python for row in csv.reader(fileinput.input(), delimiter='\t'): if len(row) > 0: print row[0] ```

That took a good 3 minutes to execute on my laptop.

Since I’m used to UNIX data processing, I tried cut -f1. Weirdly, that’s worse. 5 minutes. Paradoxically, awk '{print $1}' only takes 17 seconds. That's about 12 times faster. Clearly the tool makes a big difference. And we always knew UNIX was fast.

But I also ran these on an Amazon EC2 server, and a Hostgator server. Here’re the results.

	python	cut	awk
My Dell E5400	3:04 (1x)	5:42 (0.5x)	0:17 (11x)
EC2 standard	0:33 (6x)	0:5.6 (33x)	0:16 (11x)
Hostgator	0:19 (10x)	0:2.5 (74x)	0:0.7 (265x)

What took 3 minutes with Python my Dell E5400 took less than a second on Hostgator’s server with awk. Over 250 times faster. (Not 250%. 250 times).

And it’s not just hardware. A good tool (awk) made things 11x faster on my machine. Good hardware (hostgator) made the same program 10x faster. But choosing the right combination can make things go faster than 11 x 10 = 110 times. Much faster.

There are a few of things I’m taking away from this.

Good hardware can speed you up much as (or more than) choosing the right tool.
Good hardware can be rented. From many places. Cheaply.
Always test what’s fast. awk’s fastest on my machine and Hostgator, but not on EC2.

The ubiquitous Peter Weinberger

Fri, 14 Feb 2003 12:00:00 +0000

The ubiquitous Peter Weinberger. A man whose face was etched on CDs, circuit boards, walls, tables, and even watertowers. via RobotWisdom