python on S Anand

Proving Code Works with Z3

Mon, 22 Jun 2026 20:15:22 +0800

At the PyCon SG Education Summit today, Melvin’s lighting talk on “Writing Proofs in Python” began with a subtle bug in this mid-point calculation (often used in binary search or sort) in languages like Java, C/C++, Go, etc.

low = ...
high = ...
mid = (low + high) / 2

Since the integers are fixed-width, this triggers an overflow when low + high exceeds the maximum integer value.

Even popular libraries like Pandas had this bug until 2019. In fact, even Python’s native list.sort() had this sort of bug until 2015! Read the details.

Z3 can find counterexamples

Melvin showed how to use Z3 - a popular theorem prover library - to prove or find errors in code.

from z3 import *

low, high = BitVecs("low high", 32)
s = Solver()
s.add(low >= 0, high >= 0, low <= high)
mid = (low + high) / 2   # signed 32-bit arithmetic

# Find a case where the "obvious" midpoint is wrong
s.add(mid < low)

print(s.check())
print(s.model())
# Prints something like [high = 2142306236, low = 47120451]

The output has found a case where the midpoint is less than low, which is clearly wrong.

ChatGPT can run Z3 to find issues

I had a conversation with ChatGPT that ran like this:

Me: Prove something with Z3 and teach me.
ChatGPT: OK. Here’s how you find the midpoint overflow error.
Me: How can Z3 help me?
ChatGPT: Find edge cases, counter-examples, refactor safely, validate rules, write tests, model workflows, etc.
Me: Any alternatives to Z3?
ChatGPT: Z3 is best, but consider Dafny for production code, Alloy for data, TLA+ for workflows, …
Me: Any popular Python tutorials have Z3-detectable bugs?
ChatGPT: Several! GeeksforGeeks’ Interpolation search hits a ZeroDivisionError; in Programiz’ Bucket sort, the bucket index can exceed the number of buckets, …
Me: Find production Python code with Z3-detectable bugs.
ChatGPT: Sure! There are plenty of ZeroDivisionErrors in networkx, rich, seaborn, xarray, plotly, matplotlib, …
Me: Find real errors - not nitpicky ones!
ChatGPT: OK, xarray, branca, mizani, shap, matplotlib, …
Me: If these don’t have existing bugs filed, write an issue.
ChatGPT: Sure! Here they are.

Some issues are not bad

One issue was quite relevant. I raised #222 on python-visualization/branca which roughly says:

LinearColormap(["red", "blue"], vmin=0, vmax=1).to_step(n=1) fails. But there are times when I just have one bucket/class to display, so shouldn’t this work?

Another looked pretty relevant, too. I raised #11397 on pydata/xarray which roughly says:

np.linspace(0, 1, num=1) works. Shouldn’t RangeIndex.linspace(0, 1, num=1, dim="x") do the same?

But not all issues are worth reporting

Maybe not. The branca LinearColorMap issue feels more real to me than the xarray RangeIndex.linspace one - maybe because I’ve faced it.

Maybe we could do this:

When you find a bug, use Gen AI to report it. It’s a real need.
If Gen AI finds a bug, report it only if you’ll really need it.

Still, Z3 + ChatGPT is quite effective

When testing your own code, the ability to prove it correct or find counterexamples is very powerful.

Since you don’t need to know how to use these tools (AI does it for you), the cost of using these is very low.

This adds a useful layer of defense against vibe coded technical debt.

UPDATE: 25 Jun 2026: The pull request I submitted to fix it was merged. But this led to a fascinating rabbit hole involving the Bounty-Hunting Agent Ecosystem.

Sambar Styles

Sun, 26 Apr 2026 20:21:51 -0400

My wife’s sambar tastes different from my mother’s. And mine, too. When I cooked as a bachelor, my neighbour would pop by, taste the sambar, and exclaim, “Rasam super!”

Surbhi’s Day 5 of the 30-day challenge was about Sambar which inspired me to take her dataset and create a decision tree for which state a sambar recipe is from based on its ingredients.

ChatGPT started with 68 recipes and built a tree at 41% accuracy. As we added more recipes:

Recipes	Accuracy
68	41%
293	42%
361	55%
406	54%

… the accuracy wasn’t improving all that much.

Here is the classifier script: sambar_fftree.py. You can run it via:

uv run https://files.s-anand.net/blog/2026-04-26-sambar-styles/sambar_fftree.py

But a ingredients are snipers: rare, precise, devastating.

Kokum or goda masala -> Maharashtra. 32 for 32 in one run. Perfect.
Sesame/gingelly oil -> Tamil Nadu. 28 for 32. Strong.
Coconut oil + shallots -> Kerala. 22 for 29.
Moong dal + no mustard seeds -> Andhra. 31 for 36. Better than garlic.
Byadagi chillies -> Karnataka. 6 for 7. Tiny sample, but clean.

But without some of these strong signals, the sambar could be from anywhere. Better to abstain when unsure.

Here is the classifier that allows abstentions: sambar_fftree_abstain.py and the dataset I used. You can run it via:

# Download the files
wget https://files.s-anand.net/blog/2026-04-26-sambar-styles/sambar_recipe_dataset.csv \
     https://files.s-anand.net/blog/2026-04-26-sambar-styles/sambar_fftree.py \
     https://files.s-anand.net/blog/2026-04-26-sambar-styles/sambar_fftree_abstension.py
# Run the script with the data I used
uv run sambar_fftree_abstension.py --no-download

Only about a third of recipes have a clear signal.

Incidentally, Coconut alone is not a Kerala signal. It’s more “west coast”.

State	Uses grated coconut
Karnataka	60%
Kerala	58%
Maharashtra	32%
Tamil Nadu	24%
Andhra	14%

Garlic is not Andhra either. In one run, garlic + no coconut was a 50-50 split between Andhra and Maharashtra.

Rather than states, it’s better to think of styles.

Style	Ingredients
Amti	kokum, goda masala
Tamil tiffin-sambar	sesame oil, sambar powder, tamarind
Kerala coconut-tempering	coconut oil, shallots
Andhra pappu/charu	moong dal, less mustard
Karnataka sweet-roasted	byadagi, jaggery, coconut

Maharashtra is easy to identify if it’s amti style. Without kokum or goda masala, it’s generic sambar.

Tamil Nadu sambar has two distinctive styles: sesame/gingelly oil; or sambar powder + tamarind.

With all of this, we could identify the state only about one-third of the time based on ~2,000 recipes. But we can identify the distinctive styles from their ingredients, when it’s present.

Like my bachelor-days sambar, which was missing dal. (No one told me sambar needs dal.) And my neighbour could identify it instantly. As rasam.

Things I Learned - 18 Jan 2026

Sun, 18 Jan 2026 00:00:00 +0000

This week, I learned:

Vulture is a neat library that funds unused Python code. uvx vulture script.py works fairly well, out-of-box. This helps when cleaning up AI-edited scripts that often have left-over code or imports.
One of the lightest alternatives to Google Analytics is GoatCounter. If you just want page views, referrers, browsers, OSes, countries, and devices, it’s great. It’s privacy-friendly (no cookies), open source, easy to self-host, free for small sites, and the data is exportable.
The number of countries that allow visa-free entries to Indian passports is gently growing in Asia (Kazakhstan, Thailand, Sri Lanka, Malaysia, Iran, and Philippines).
Lessons from performance books. Claude # #
- Summary: In early days, explore, sample. Then narrow based on interest & fit. Practice hard and persist.
- ⭐⭐⭐⭐ Range (David Epstein): In changing environments (rules shift, feedback is noisy/late), sample broadly, i.e. generalize. Specialization vs generalization
  - Nobel laureates have more hobbies. Olympic athletes have less. Shift nurses have same hobbies as non-shift workers. Hobbies help expertise in some areas
  - Rewarding ONLY what succeeds locks behavior, halts exploration. Vary / delay incentives. Reward AFTER figuring out what works. Reinforcement and rewards
  - Maybe “orderly” people specialize and creative people generalize? So pick what aligns with personality?
- ⭐⭐⭐ Peak (Anders Ericsson & Robert Pool): Compounded practice at the edge of competence, with good immediate feedback, helps 14-26%.
  - But talent (genetics, upbringing, brainpower) differentiates more the expert level.
  - Slow, effortful practice (spaced recall, interleaving topics, self-testing) builds lasting knowledge - but looks inefficient and doesn’t help with exams. Learning and long-term retention
  - “Easy” 10K hours don’t help.
- ⭐⭐ Grit (Angela Duckworth): predicts roughly the same as conscientiousness (18%). It predicts success in stable paths moderately (but brainpower, etc. matter too).
  - But premature grit hurts. Quit if it helps.
  - But environment can defeat grit.
Lessons from attention economy books. Claude # #
- The attention economy is real. It is designed to capture our mind, and it is winning.
- Distractions hurt MUCH more than we think. Batching, focus time helps.
- Privilege helps. The rich have more control over these than the poor do.
- ⭐⭐⭐⭐ Deep Work (Cal Newport, 2016) and ⭐⭐⭐ Digital Minimalism (Cal Newport, 2019): control the tools. Focus time, digital detox, embrace boredom. This helps - when you can afford to.
- ⭐⭐⭐ Indistractable (Nir Eyal): control yourself. The problem is internal (also true), so build habits, since willpower depletes (hm… not really).
- ⭐⭐⭐ How to Do Nothing (Jenny Odell, 2019): reject. Embrace boredom as resistance. This helps - when you can afford to.
- ⭐⭐ Stolen Focus (Johann Hari, 2022): regulate & rebel. The problem is systemic and external (also true). Reclaim your interface.
- BTW: Goldfish have excellent attention spans and memory :-)
Lessons from trauma books. Claude # #
- ⭐⭐⭐ The Body Keeps the Score (Bessel van der Kolk, 2014): trauma recall shuts down the speech area. Eye movement desensitization (EMDR) helps. So does CBT, despite what the book says. But does yoga (only a little) or neurofeedback (too little data)?
- ⭐⭐⭐ What Happened to You? (Bruce Perry & Oprah Winfrey, 2021): calming people down before talking. Strong connections help more than a therapist.
- ⭐⭐ The Myth of Normal (Gabor Maté, 2022): trauma causes cancer (no), autoimmunity (partly), ALS (?), etc.
- ⭐ It Didn’t Start with You (Mark Wolynn, 2016): maybe anxiety is epigenetic and heriditary? Unproven. Family Constellation Therapy is wrong
- ⭐⭐ My Grandmother’s Hands (Resmaa Menakem, 2017): maybe racism is a somatic (body) response to generational (epigenetic) trauma? Too little data
- ⭐⭐ No Bad Parts (Richard Schwartz): maybe we’re not one person but a collection of parts, and interviewing family systems (IFS) helps? Unclear
- ⭐⭐⭐ Maybe You Should Talk to Someone (Lori Gottlieb): our memory is unreliable and therapy is messy. Connection & compassion help
- Most of these are based on the contested Polyvagal Theory: the nervous system scans for danger before the mind can process it. But the specific claims of the theory are wrong and it makes no other falsifiable claims.
  - The nervous system has hierarchical responses to threat. 🟢 Not unique to PVT
  - Social connection regulates physiology. 🟢 Not unique to PVT
  - Unconscious threat detection (neuroception). 🟡 Weak evidence
  - Mamellian brain (ventral vagal system) is uniquely mammalian. 🔴 Lungfish have it
  - Reptilian brain (dorsal vagal) “shutdown” causes dissociation. 🔴 No evidence
  - RSA directly measures vagal tone. 🔴 Contested
  - Reptiles are “asocial”. 🔴 Wrong
- Trauma causes body changes too. It’s not just the mind.
- Childhood trauma persists.
- Relationships (connection & compassion) help more than therapy
What constitutes tax residency in India? For an Indian citizen, as I understand it (after 2 hours of research):
- If you were in India >= 182 days: Resident*
- Else, if you left India this year for employment: NRI.
- Else, if you are an Indian Citizen living abroad (visiting or not):
  - If Indian Income <= ₹15 Lakhs: NRI.
  - Else if you were in India >= 120 days AND >= 365 days in the last 4 years: RNOR.
  - Else if you are not liable to tax in any other country: RNOR.
- Else, if you left India for non-employment (students, tourism) and were in India >= 60 days AND >= 365 days in the last 4 years: Resident*
- Else: NRI.
- If you ended up as a Resident*
  - If you were NRI in 9 of the last 10 years OR in India <= 729 days in the last 7 years: RNOR
  - Else: ROR (Resident & Ordinarily Resident).
- For all practical purposes, RNOR is like an NRI. You pay tax only on Indian income, not global income. It’s like a transition status for returning NRIs.
AVIF compresses better than WebP and may be the “next big thing”. I will be switching for all future images. Squoosh remains my choice of compressor and Ezgif’s AVIF maker and GIF to AVIF are handy.

Finding open source bugs with Ty

Sun, 11 Jan 2026 11:55:19 +0800

Astral released Ty (Beta) last month.

As a prototyper, I don’t type check much - it slows me down. But the few apps I shipped to production had bugs type checking could have caught. Plus, LLMs don’t get slowed by type checking.

So I decided to check if Ty can spot real bugs in real code.

I asked ChatGPT:

Run ty (Astral’s new type checker) on a few popular Python packages’ source code, list the errors Ty reports (most of which may be false positives), and identify at least a few that are genuine bugs, not false positives. Write sample code or test case to demonstrate the bug.

Don’t fool yourself into inventing false or borderline bugs. Find genuine bugs.

After 25 minutes, ChatGPT reported:

from tqdm.rich import tqdm

bar = tqdm(total=3)
bar.update(1)

# This crashes inside tqdm's rich integration
bar.reset(total=3)

… produces this error:

TypeError: Progress.reset() missing 1 required positional argument: 'task_id'

I verified this locally:

uv run --with tqdm python -c '
from tqdm.rich import tqdm
bar = tqdm(total=3)
bar.update(1)
bar.reset(total=3)'

Yes! Same error.

But is it really a bug?

I asked Gemini:

Is this a real tqdm bug?
- Yes, it is.
Could I submit a pull request? Is tqdm active and reviewing/accepting pull requests - especially for the rich integration?
- Yes, you should.
Find existing PRs with overlapping functionality. List the closest ones
- Issue #1306 and PRs #1395, #1438 are close, but different.
(I checked manually.) #1674 looks very similar… is it?
- No, that’s a different bug
(I wanted to be sure.) What about this issue: #1378
- Yes, that’s an exact duplicate, waiting since 2022. (Stupid LLM!)
What is the minimal fix?
- Add self._task_id as the first argument to self._prog.reset(total=total) in tqdm/rich.py
This PR seems to do that: #1379 doesn’t it? Why is it not merged yet?
- Yes, it was even approved, but not merged. Bump up. Else, submit your own PR.

So I bumped it up

This entire process of:

Finding a bug
In a new codebase
Verifying the bug
Finding an existing PR
Bumping it up

… took 30 min of my time and 30 min of computer time.

The hardest problem in computer science isn’t P-vs-NP or cache invalidation. It’s understanding why a human will click a green ‘Merge’ button.

Get this blog via email on Google Groups

Mon, 05 Jan 2026 17:50:35 +0800

TL;DR: Join this Google Group to get my blog updates via email.

My blog is over 25 years old. At first, people had to visit it to read it. Then I added an RSS feeds. Then email subscriptions. Then via social media, cross-posting on Twitter, now LinkedIn.

The RSS feed remains. But I feel it’s time to bring back email subscriptions. It’s the oldest of the technologies, the most robust, and the one I believe will last the longest.

My blog might last another 25 years. I’m willing to bet email will outlast RSS and certainly the social media platforms.

So, I’ve created a Google Group: s-anand@googlegroups.com where I will post new blog entries. You can join the group to get email updates whenever I post something new.

For now, I will post updates manually. After a few weeks of testing, I’ll automate the process.

PS: I use a htmlemail.py script to send blog posts as HTML emails via the GMail API. It was vibe-coded by Claude and taught me a few things:

The frontmatter package is a clean way to parse YAML frontmatter from markdown files.
LLMs aren’t familiar enough with the markdown2 package, which is the new standard for processing Markdown. They prefer markdown instead.
premailer is a well-maintained package to inline CSS for HTML emails.
pygments is the de facto standard to generate CSS for syntax highlighting code blocks.

Thu, 20 Nov 2025 00:00:00 +0000

While meditating, I realized 75% of “LULL” is the letter “L”. (This sort of thing happens a lot when I meditate.)

MUMMY (60% M) and DADDY (60% D) have lower percentage, but are longer, so maybe get a bonus?

I asked Claude Code what would top such a list. It picked a dictionary, generated the 333 words with 4+ letters and >50% concentration.

What did I like best? “ASSESSES”. 5/8 letters are “S”. That’s nearly two-thirds.

Then it wrote an xkcd-style story! Here are snippets:

I wrote a Python script to analyze 370,105 English words. Then I questioned my life choices.

*Does “mmmm” count? Look, I didn’t make the dictionary. I just downloaded it from the internet.

Notice something? After length 6, it’s just S-words and E-words all the way down. (It’s the grammatical duct tape of English)

I didn’t analyze Welsh, but I’m guessing their L’s would be extremely competitive.

The story way more fun than the analysis!

Check it out: https://github.com/sanand0/research/tree/main/repeated-letter-words

Learning: Find good styles to copy from. Here’s my style catalog: https://github.com/sanand0/prompts/blob/main/styles.md

Things I Learned - 29 Jun 2025

Sun, 29 Jun 2025 00:00:00 +0000

This week, I learned:

“People are great at feedback on what you are doing wrong. They are not so good at telling you how to fix it. They don’t know you that well.” Amit Kapoor
Perfect Cursors makes periodic cursor positions animate smoothly by interpolating on a spline**
CloudFlare and Vercel now support sandboxes where you can execute code. The price is not so low that we can execute for free in bulk but works well infrequent or batched code execution. Simon Willison
Here’s how I’m using ffmpeg for video recording & editing.
- To record screen at 5 frames per second, I run an abbreviation screenrecord which maps to:
Gemini CLI has a generous free tier and uses Bootstrap over Tailwind Ref #ai-coding
Cloudflare has a native agents SDK that looks good, especially for CloudFlare users. Ref
There are several brands with recognizable chart style guides. It’s possible to generate style guides for these from the charts, but applying them via matplotlib is almost #impossible today. ChatGPT
Hyperfine is like %timeit for the shell. Written in Rust
⭐ Vertical AI is a moat against AGI. Specialization reduces hallucinations. Custom workflows and regulations are sticky and defensible. We need to start selling to users, not IT, though. Ref
When AI automates a task, the bottleneck shifts. AI process re-design is about reworking the process around the new bottleneck, and iterating quickly.
- With coding, it’s testing, reviewing, deploying, use-case identification.
uvx git-smart-squash re-organizes haphazard commits using LLMs. git-smart-squash #ai-coding
GitHub offers a free Docker container registry. Simon Willison
There are three major areas where humans either are, or will soon be, more necessary than ever: trust, integration and taste – NYT. Anil. To deal with this:
- Learn things that might grow in importance, like:
  - Data modeling
  - APIs
  - Code reviews
  - Drawing and 3D modeling
  - Narrative storytelling
  - Design
  - Movie making
  - Statistics
  - Sceptical fact checking
  - Continuous AI auditing e.g. awesome-continous-ai or automated-auditing
  - Zero knowledge proofs
  - Homomorphic encryption
  - Privacy-preserving computation
  - Fingerprinting and watermarking
  - Governance frameworks
  - Ethics and AI dilemmas
  - Negotiation
  - Change management
  - Remote working, management, hiring
  - Creating attention scarcity
  - Local cultures
- Work with people of growing importance
  - People designing products in regulated industries
  - Cross domain experts
  - Art developers, game makers, designers
  - System thinkers. Economists, ecologists, system planners. People who look for second order effects.
- Live in cities that might play a bigger role in the future
  - Cities like Singapore and learn how it builds civics trust, creates digital IDs.
  - Cities like Bangalore and Hyderabad and learn how they grow tech talent
  - Creative cities like Paris, Seoul, Mexico City, Berlin, etc. on sabbaticals to taste hubs
- Try to:
  - Build auditing credentials and IP
  - Audit your calendar for what AI can do. Have it interview you
  - Practice sceptical fact checking and audit
A clever way to test a library’s quality is to have LLMs write code from docs and test it. Failing libraries have flawed code/docs. Improve. Ref #ai-coding
Common Pile is an 8TB open dataset for LLM training that includes ArXiv, PubMed, StackExchange, GitHub, IRC, Regulations.gov, Patents, UK parliament, books. Easier than scraping.
A useful way to have reasoning models do deep-research-like work is to have them “First, create a plan to solve the problem, clearly listing the objective, approach, and output. Then follow the plan.”
DE-COP is a method to check if LLMs were trained on private content. GPT-4o was trained on O’Reilly books, based on this method. Ref
LLMs are more persuasive than humans. But repeated exposure reduces the effect. Ref
Phoenix.new uses live views to publish apps as it codes. The testing framework looks at the screen while it codes and fixes errors. It commits every change
Anthropic system prompt asking Claude to pursue its goals led to self preservation behavior. Ref
The hungrier I am the better the food tastes. A good reason to eat less quantity and frequency
You can purge the jsDelivr cache manually. Helps if you released a new version of a package and way to purge an alias (e.g. https://cdn.jsdelivr.net/npm/your-package@1)
XConvert is a convenient online app to compress .webm videos. Not great design but fairly good compression.
You can draw a treemap of import times via python -X importtime app.py > timing.txt and then paste them at https://kmichel.github.io/python-importtime-graph/.
PyOpenLayers adds interactive mapping via OpenLayers to Marimo and Jupyter.
In a TechCrunch interview with Jared Kaplan has was asked if Anthropic is becoming less safety conscious because they released Opus 4 which blackmails. Kaplan replied that they have stronger testing and higher transparency, so they’re more likely to share AI dangers early. Great positioning! Conversations are about perspective change and this nailed it.
The system prompts for Anthropic misalignment evals are a fascinating read.
AI PR Watcher tracks GitHub pull requests from Codex and other LLMs. Codex is way ahead of anything else on volume and success rate. Devin is next on volume, Cursor is next on success rate.

Things I Learned - 27 Apr 2025

Sun, 27 Apr 2025 00:00:00 +0000

This week, I learned:

OpenAI’s reasoning models are much ahead of other models when multiplying two numbers in their heads. Ref
⭐ Promptfoo may be the most mature open source LLM evals tool. Simon Willison
Dyson Sphere.
LemonSlice showcases real-time audio-video models (avatars) that are close enough to real.
Notes from Latent Space ICLR 2025, Singapore
- Daniel: Menlo’s ReZero. A model that keeps searching till it finds the answer.
  - There are multiple search techniques: Multi-step retreival, Iterative retrieval, Query rewriting. Also, reasoning.
  - The LLM token generation sequence is normally: <think>, <search>, <answer>.
  - Insight: “If we explicitly reward LLMs for retrying after a failed search, they out-perform one-attempt systems.” So <think>, <search>, <think>, <search>, <think>, <search>, <answer>.
  - ⭐ Prompt reasoning models, e.g. “Keep searching till you find the best answer.”
- Roger, Nous Research
  - Supervised learning is limited because accuracy is piece-wise linear, i.e. it’s broken up. Continuous optimization is meaningless.
  - Reinforcement learning works better because rewards can be discrete. (But it converts things back into differentiable loss functions behind the scenes.)
    - Rewards can be good/bad. Single or multi-step. Whatever.
  - We’re in the “Era of experience”, i.e. models gain experience from the environment themselves.
  - ⭐ So, we need environments models can learn in. This is the next thing after training data. That needs a standard for environments.
  - We’d need a model, a trainer, and the environment.
  - The environments whatever capabilities. Run code. Browser. A game. … With an exposed interface
- Eugene Cheah (Featherless.ai)
  - Transformer architectures need n-square GPUs as # of tokens grow. Featherless is exploring an RWKV architecture that scales linearly. THere are other such architectures. Performer, Linformer, Reformer, Hyena.
  - Mistral-Nemo-12b-ic is one of the most popular fine-tuned model. It’s small enough to run on a server.
- Justus Mattern (Prime Intellect)
  - Intellect-2 is a continously learning (RL) model that uses decentralized training on peer-to-peer GPUs.
  - Solving problems on bandwidth, verifiable contributions, etc.
ChatGPT Deep Research now also has an O4-Mini version to serve smaller reports. Free users get 0 original + 5 lightweight 5 tasks / month. $20 version gets 10 + 15. $200 version gets 100 + 150. The month begins on first use of Deep Research and runs on a 30 day “window”. Ref
O4-Mini-High is great at going through an under-documented repo and finding things. For example, here’s how I configured cmdg.
ChatGPT is my new Jupyter Notebook :-)
Google announced new AI capabilities at Google Next APAC 2025. Blog. Interesting ones are:
- @Gemini in chat
- Google Meet support for “Catch me up”
- Google Vids: Create short video clips
- Google Sheets: does better analysis
- Google Slides: image generation
- Google Docs: Create Audio Clips (like NotebookLM in Google Docs)
- Google Docs: “Help me refine” is better than before
- Google Workspace Flows
gcalcli is a convenient way to export Google Calendar. Example: uvx gcalcli agenda --tsv 2025-01-01 2025-01-05
cmdg is a command line GMail client that I’ve now switched to for quick email checks. 80% of my email is spam and this is good enough to scan and delete those. It also avoids running a 200-500 MB tab in the browser that constantly shows me how many unread emails I have.
From Worklife with Adam Grant: Cancelling cancel culture with Loretta Ross
- “Lighten up! Fighting Nazis should be fun. It’s being a Nazi that sucks. If you’re not having fun fighting for hope and joy and human rights, maybe you’re doing the fight wrong. We are the ones who should be having fun.”
- “You can say what you mean. But you don’t have to say it mean.” There is always a way to put it across better. Refusing to say mean things is about to discover these approaches.
- “The true mark of a lifelong learner is knowing that you can learn something from every single person you meet.” If you remember that, you can’t be a know it all.
semantic-text-splitter could be the go-to text splitter. It’s Rust-based, supports MarkdownSplitter, and multiple tokenizers. Alternatives like semchunk, advanced-chunker, chonkie, etc. seem clunkier.
ULID is like UUID but time-sortable. That’s an improvement over timestamp IDs (definitely) and potentially even UUIDs. They can be generated by clients as a globally unique ID. Try pip install python-ulid and npm install ulid.
The Consumer Product Safety Commission Data has thousands of reports of product safety over time
You can run xclip -sel clip -o | pandoc -f markdown -t html --no-highlight | xclip -sel clip -t text/html -i to convert Markdown in the clipboard to rich text. But xclip doesn’t support multiple selections, so the text is lost. ChatGPT
DuckDB UI & Notebooks will potentially be a good alternative to Datasette, DBeaver, etc. But for now, there are still glitches. It crashes with a SIGSEGV (Address boundary error) when connecting to SQLite databases.
Ollama limits MAX_TOKENS to 2K by default.
AI assisted search helps wherever I would have used Google, e.g.
- Debugging. “Fix CUDA initialization: CUDA unknown error”
- Tool search. “Find an online word counter tool.”
- Library search. “Find a JS micro library to render Markdown.”
OpenAI API capabilites lag ChatGPT features. For example:
- o4-mini via the API does not search the web natively as part of its reasoning.
- o4-mini, o3, o3-mini, o1, gpt-4.1-nano don’t yet support the web_search_preview tool. Only gpt-4.1 and gpt-4.1-mini do. Limitations
- Search results are NOT visible via the API. They’re fed directly to the model. The number of searches or results is unknown. Each search costs 0.25-0.5 cents. Pricing
- For reasoning traces (e.g. .reasoning.summary: "medium") you need to verify your organization via withpersona.com which failed with my Indian passport AND Singapore work permit.
The ChatGPT Plus plan ($20) gives you 50 O4 mini messages a day, which I exceeded! It’s supposed to reset at midnight UTC Ref but might operate on a rolling window ChatGPT. “Currently, there is no way to check how many messages you have used in your usage budget.” OpenAI
SignalBloom reads SEC filings and writes analyst reports on it using LLMs
“Evaluation in the loop” or “Evals-in-the-loop” is a new term I learnt. SignalBloom’s Hallucination Bechmark
If AI interacts with the world and generates data from its own experience and learns from that, we have a new scaling mechanism. DeepMind podcast
OpenAI’s search API is fairly expensive at $30+/1K calls. Typically, to read interesting HN articles, I will make 30 calls which is about 75c. Instead I should use the app and summarise HM news across different days manually based on my interests!
Finally! t-strings land in Python. They’re like JavaScript template literals.
DuckDB’s CSV parser might be one of the most forgiving parsers. Even better than Pandas or SQLite3. Ref
Good managers will probably make good AI managers. AI agents can probably substitute humans in business experiments. Ethan Mollick
If Windsurf stops working, reload the extension. GitHub
TLS certificates will start expiring in 47 days from 15 Mar 2029, forcing automated domain renewals. Digicert
Nix flakes are a reliable alternative to DevContainers that don’t need Docker - but don’t work on Windows.
Ink is like React for the CLI.
The Unsure Calculator is a great tool to calculate formulas with multiple uncertainties, like:
- My office is 9-11 km away and it takes me 45-55 min to reach. So I cycle at 9~11 / 45~55 * 60 ~ 10-14 kmph (12 most likely).
- I spend $6-15 on lunch and eat out 80-120 days a year. So I spend 6~15 * 80~120 ~ $600~1550 ($1000 most likely) eating out yearly.
- I take 30-120 min to prepare a quiz question. Each exam has 6-12 questions. So I need 30~120 * 6~12 / 60 = 4~20 hours (11 most likely)
Using Kiran’s macOS setup for dev I enabled colorized less and mouse options for tmux.
time fish -i -c exit prints the time taken for fish startup. fish --profile-startup ~/fish.profile -i -c exit prints the time taken by each command on fish startup to ~/fish.profile. I used this to speed up my fish startup.
The 8 top features of the OpenAI Responses API that are an improvement over the Completions API (IMHO) are:
- Link to previous response rather than sending history
- Uploading files directly
- Swappable system instructions while retaining the chat history
- Customisable reasoning effort AND reasoning summary detail
- Truncation in the middle option
- Web search context size option
- File search filters by file attributes
- Flex service tier for lower cost
OpenAI doesn’t charge for file storage but does charge 10 cents / GB-day for vector storage beyond 1 GB. The first 1GB is free
Augment Code is an AI code editor that’s growing popular on Reddit. #ai-coding
The GPT 4.1 models have a 75% discounted prompt caching (instead of the usual 50%), making them particularly suited for repetitive tasks. OpenAI
chatgpt.com shortcut keys are revealed via Ctrl + /. Here’s my ranking on usefulness:
- Ctrl + Shift + C: Copy last response as Markdown!
- Ctrl + Shift + ;: Copy last code block
- Ctrl + Shift + S: Sidebar toggle
- Ctrl + Shift + O: Open new chat
- Shift + Esc: Focus chat input
- Ctrl + Shift + I: Ccustom instructions
- Ctrl + Shift + X: Delete chat

Things I Learned - 02 Mar 2025

Sun, 02 Mar 2025 00:00:00 +0000

This week, I learned:

Proxmox Virtual Environment is an open-source alternative to VMWare, Hyper-V, Citrix XenServer, etc. (There’s nothing there that prompts me to explore it further.)

With Podman on Windows (a Docker equivalent), many Docker-enabled tasks become easier. For example, running PostgreSQL is as easy as:

podman run -d --name postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:latest
podman exec -it postgres psql -U postgres -c "CREATE DATABASE mydb;"

Bad deep research prompts are: vague/broad, under-specified or ambiguous. In short, the more you know what you want, the better. Iterate until then.
What kind of reports do clients are research companies to produce? I was curious to see if Deep Research can replace these. Here are a bunch of ideas. ChatGPT
1. Strategy & Management Consulting Research (McKinsey & Company, Boston Consulting Group, Bain & Company, Strategy&, Accenture Strategy)
  - Produce a comprehensive strategic transformation report for a Fortune 500 consumer goods company. Analyze global market trends, competitor strategies, and actionable growth recommendations, including case studies and source citations.
  - Generate an in‐depth study on corporate restructuring trends in emerging markets. Focus on successful turnaround strategies, CEO leadership factors, and strategic pivots, with a comparative analysis of key players.
  - Create a report on M&A trends in the technology sector over the past five years. Detail deal drivers, integration best practices, and forecast future acquisition opportunities, citing relevant data.
2. IT & Technology Research Analysts (Gartner, Forrester Research, IDC, 451 Research, Ovum)
  - Produce a market assessment report on emerging cloud computing platforms. Include vendor evaluations, adoption forecasts, and key technology drivers with supporting data and charts.
  - Generate an in‐depth cybersecurity trends report for enterprise IT. Analyze recent threat vectors, defense strategies, and best practices for risk mitigation, providing actionable recommendations.
  - Create a comprehensive study on the impact of artificial intelligence in enterprise software. Include competitive benchmarking, technology adoption rates, and forecasted market changes.
3. Marketing & Consumer Research (Nielsen, Kantar Group, Ipsos, GfK, Euromonitor International)
  - Produce a consumer behavior analysis report for a leading retail brand. Identify key demographic shifts, purchasing trends, and brand loyalty factors, and provide actionable insights with data visualizations.
  - Generate a detailed report on digital media consumption trends among millennials, incorporating survey results, social media analytics, and case studies of successful campaigns.
  - Create a market segmentation report for a new consumer electronics launch. Identify key consumer segments, behavioral drivers, and media usage patterns with clear recommendations.
4. Financial Investment Research (Goldman Sachs, JPMorgan Chase, Morgan Stanley, Morningstar, Keefe Bruyette & Woods)
  - Produce an equity research report on mid-cap technology stocks. Include detailed financial modeling, valuation analysis, and buy/sell/hold recommendations with supporting data and charts.
  - Generate a fixed income analysis report for corporate bonds in the industrial sector. Assess credit risk, yield forecasts, and macroeconomic influences, citing key data sources.
  - Create a comprehensive report on global market trends impacting investment banking. Analyze regulatory changes, market sentiment, and performance metrics of leading financial institutions.
5. Healthcare Research (IQVIA, Frost & Sullivan, Evaluate Ltd, Deloitte Healthcare, IMS Health)
  - Produce a market analysis report on emerging biotechnologies in oncology. Include competitive landscape, regulatory challenges, and growth forecasts with relevant case studies.
  - Generate a comprehensive report on patient satisfaction and telemedicine adoption trends. Analyze survey data from leading healthcare providers and benchmark best practices.
  - Create a detailed study on pharmaceutical market dynamics in emerging economies. Focus on pipeline developments, regulatory environments, and market potential with actionable insights.
6. Legal Research Providers (LexisNexis, Westlaw, Bloomberg Law, Fastcase)
  - Produce a legal risk assessment report on the impact of recent data privacy regulations for multinational corporations. Include case studies, trend analysis (2019–2024), and strategic recommendations.
  - Generate a comprehensive report summarizing key federal and Supreme Court rulings on intellectual property rights over the past five years, highlighting trends and divergent interpretations.
  - Create a detailed report on the evolution of securities law and its effect on investment research practices, incorporating analysis of recent litigation and regulatory updates.
7. Media & News Research (Factiva, Kantar Media, Comscore, Cision)
  - Produce a media consumption trends report that analyzes audience behavior shifts across digital, TV, and print platforms. Include data visualizations, key drivers, and forecasted trends.
  - Generate a comprehensive report on the impact of social media on traditional news reporting, with case studies and a comparative analysis of engagement metrics.
  - Create a detailed study on the effectiveness of multimedia advertising campaigns, evaluating ROI, consumer engagement, and best practices with actionable insights.
8. Economic & Industry-Specific Research (Economist Intelligence Unit, BMI Research, IHS Markit, Consensus Economics)
  - Produce a macroeconomic outlook report for emerging markets, including GDP, inflation, and employment forecasts, with detailed data analysis and visualizations.
  - Generate an industry analysis report on the automotive sector, covering technological innovations, competitive dynamics, and consolidation trends.
  - Create a comprehensive country risk assessment report for a target region, detailing political, economic, and regulatory factors with recommendations for investors.
9. Human Resources & Employee Engagement Research (Gallup, Great Place to Work, Mercer)
  - Produce an employee engagement report for a multinational firm based on recent survey data. Identify key drivers of satisfaction, retention challenges, and improvement recommendations.
  - Generate a comprehensive study on the impact of remote and hybrid work models on employee productivity across industries, including best practices and benchmark data.
  - Create a detailed report on workplace culture transformation, analyzing organizational behavior trends, employee feedback, and actionable strategies to boost engagement.
10. Environmental, Social & Governance (ESG) Research (MSCI ESG Research, Sustainalytics, ISS ESG, Bloomberg ESG)
  - Produce an ESG performance report for a portfolio of global companies. Include sustainability scores, risk assessments, and recommendations for improvement with data visualizations.
  - Generate a comprehensive study on the impact of climate change regulations on the energy sector, including policy analysis, market forecasts, and strategic implications.
  - Create a detailed report on corporate social responsibility trends in the consumer goods industry, incorporating qualitative and quantitative analyses with actionable recommendations.
11. Education & Academic Research (RAND Corporation, National Center for Education Statistics, HolonIQ)
  - Produce an analysis report on the future of online education, examining technological adoption, market growth projections, and student outcome trends with supporting data.
  - Generate a comprehensive study on the effects of educational policy reforms on public school performance in the U.S., including trend analysis and actionable recommendations.
  - Create a detailed international higher education trends report, covering tuition dynamics, international student mobility, and emerging academic programs with comparative data.
12. Real Estate & Property Research (CBRE, JLL, CoStar Group, Cushman & Wakefield)
  - Produce a commercial real estate market analysis report for major urban centers, including occupancy trends, rental rate forecasts, and investment opportunity assessments.
  - Generate a comprehensive study on residential housing market dynamics in emerging economies, focusing on affordability, supply-demand gaps, and policy impacts.
  - Create a detailed report on the impact of urban redevelopment projects on local real estate values, including case studies, forecasts, and strategic recommendations.
13. Energy & Natural Resources Research (Wood Mackenzie, Rystad Energy, Bloomberg New Energy Finance)
  - Produce an analysis report on global renewable energy trends, covering technology adoption, market forecasts, and key policy drivers, with detailed data and visuals.
  - Generate a comprehensive commodity price forecasting report for oil, natural gas, and key metals, incorporating historical trends, risk assessments, and predictive modeling.
  - Create a detailed report on energy transition strategies for traditional energy companies, focusing on clean technology investments and market adaptation strategies.
14. Supply Chain & Logistics Research (ARC Advisory Group, Gartner Supply Chain Research, Supply Chain Insights)
  - Produce a report on supply chain resilience for global manufacturers. Analyze risk factors, digital transformation impacts, and best practices for operational efficiency with supporting data.
  - Generate a comprehensive study on the impact of technology on logistics networks, including case studies on digital optimization and cost reduction strategies.
  - Create a detailed report on emerging last-mile delivery solutions, assessing innovations, consumer expectations, and scalability with actionable insights.
15. Cybersecurity & Information Security Research (KuppingerCole, Forrester Security, IDC Cybersecurity, Cybersecurity Ventures)
  - Produce an in-depth report on emerging cybersecurity threats for large enterprises, including detailed analysis of recent incidents, risk vectors, and defense strategies.
  - Generate a comprehensive cybersecurity market landscape report, evaluating vendor performance, technology forecasts, and best practices for mitigating risks.
  - Create a detailed report on regulatory compliance trends in information security within the financial services industry, with case studies and strategic recommendations.
16. Social Media, Digital & Online Research (Comscore, SimilarWeb, Brandwatch)
  - Produce a digital audience behavior report for a global brand, focusing on social media trends, engagement metrics, and platform performance with detailed data analysis.
  - Generate a comprehensive analysis of influencer marketing effectiveness across digital channels, including ROI metrics, case studies, and best practices.
  - Create a detailed report on online brand sentiment analysis, incorporating social listening data, trend forecasts, and actionable recommendations.
17. Public Opinion & Political Research (Pew Research Center, Gallup, YouGov)
  - Produce a public opinion polling report on voter sentiment ahead of a major election. Include demographic breakdowns, key issue analysis, and trend visualizations for the past five years.
  - Generate a comprehensive study on political risk in emerging markets, analyzing historical data, current trends, and future projections, with policy recommendations.
  - Create a detailed report on the influence of media on public policy, using survey data, social media analysis, and comparative case studies.
18. Sports, Entertainment & Media Research (Nielsen Sports, Sportcal, Kantar Media Sports)
  - Produce a market analysis report on sports sponsorship trends, detailing viewership metrics, brand engagement, and investment ROI with industry case studies.
  - Generate a comprehensive report on audience behavior in the streaming media industry, including demographic insights, consumption trends, and competitive benchmarks.
  - Create a detailed analysis of digital advertising effectiveness in the entertainment sector, including segmentation data, ROI analysis, and strategic recommendations.
19. Innovation, R&D & Technology Trends Research (Innosight, Frost & Sullivan Innovation, CB Insights)
  - Produce a global R&D investment trends report, analyzing technology spending, innovation indices, and the impact on market growth across key industries.
  - Generate a comprehensive study on disruptive technologies in manufacturing, including competitive analysis, market potential forecasts, and adoption trends.
  - Create a detailed report on emerging innovation hubs worldwide, focusing on startup ecosystems, funding trends, and collaborative opportunities in technology.
20. Agriculture & Agribusiness Research (Rabobank Agribusiness Research, USDA Economic Research Service, AgFunder)
  - Produce an analysis report on global agricultural market trends, including crop yield forecasts, trade dynamics, and policy impacts, with data visualizations.
  - Generate a comprehensive study on agritech innovations such as precision farming and sustainable practices, including case studies and market forecasts.
  - Create a detailed report on the impact of climate change on food production and supply chain stability in agribusiness, with risk assessments and strategic recommendations.
21. Environmental & Climate Change Research (Carbon Trust, IHS Markit Energy Transition, Bloomberg New Energy Finance)
  - Produce a report on the economic and social impacts of climate change on urban infrastructure, including forecasting models and policy recommendations.
  - Generate a comprehensive study on national climate policies and their effects on industrial competitiveness, with detailed trend analysis and source citations.
  - Create a detailed report on corporate sustainability initiatives, assessing environmental risk management practices and providing actionable recommendations for improvement.
22. Customer Experience (CX) & User Experience (UX) Research (Forrester CX Research, Gartner CX Research, Qualtrics, Nielsen Norman Group)
  - Produce a report on customer journey mapping for a leading retail brand, identifying key touchpoints, pain points, and actionable improvement strategies with data visualizations.
  - Generate a comprehensive study on digital user experience trends for e-commerce platforms, including usability testing insights, design best practices, and conversion optimization recommendations.
  - Create a detailed report on customer satisfaction and loyalty metrics across multiple industries, integrating survey data and actionable recommendations to enhance overall CX.
23. Blockchain, Cryptocurrency & Fintech Research (Chainalysis, CoinDesk Research, Deloitte Fintech Research, CB Insights)
  - Produce an analysis report on emerging blockchain technologies and their applications in financial services, including market trends, adoption forecasts, and case studies.
  - Generate a comprehensive study on cryptocurrency market dynamics, analyzing regulatory developments, investor sentiment, and competitive landscapes with source citations.
  - Create a detailed report on fintech disruption in traditional banking, with case studies on leading startups, technology adoption, and future market forecasts.
24. Venture Capital, Startup & Private Equity Research (PitchBook, CB Insights, Crunchbase, Preqin)
  - Produce a global venture capital investment trends report, including performance analysis of high-growth startups, sector benchmarks, and emerging market opportunities.
  - Generate a comprehensive study on private equity market dynamics, covering deal flow analysis, exit strategies, and forecasted trends with supporting data.
  - Create a detailed report on emerging startup ecosystems in key regions, highlighting funding trends, investor activity, and growth potential with actionable insights.
25. Operations Research & Management Science Consulting (The Brattle Group, NERA Economic Consulting, CRA International)
  - Produce a report on optimization techniques for operational efficiency in large-scale manufacturing, including quantitative analysis, simulation models, and case studies.
  - Generate a comprehensive study on the application of predictive analytics in supply chain management, focusing on data modeling, process improvements, and actionable insights.
  - Create a detailed report on advanced quantitative modeling approaches to solve complex business problems in logistics and operations, including scenario analysis and recommendations.
26. Cultural & Social Research (Ethnographic/Sociocultural Studies) (Ipsos MORI, Kantar TNS, YouGov)
  - Produce a qualitative ethnographic study on urban consumer lifestyle trends, incorporating field observations, interviews, and cultural analysis with actionable insights.
  - Generate a comprehensive study on how cultural shifts influence global brand perception, including comparative case studies and trend analysis.
  - Create a detailed report on sociocultural dynamics and consumer behavior in emerging economies, integrating in-depth field research and actionable recommendations.
27. Economic & Demographic Research Firms (Oxford Economics, The Conference Board, CEIC Data)
  - Produce a macroeconomic forecasting report for a specific region, including GDP, inflation, and employment trends with detailed data visualizations and source citations.
  - Generate a detailed demographic analysis report for a target market, highlighting age distribution, income levels, and consumption patterns with actionable insights.
  - Create a comprehensive report on the economic impact of demographic shifts on consumer markets, with policy recommendations and trend analysis.
28. Academic & Think Tank Research Organizations (Brookings Institution, RAND Corporation, Carnegie Endowment for International Peace)
  - Produce a policy research report on global governance challenges and their implications for economic development, including case studies, literature reviews, and expert interviews.
  - Generate a comprehensive study on social inequality and its effects on public health and education outcomes, supported by empirical research and trend analysis.
  - Create a detailed report on emerging trends in international relations and their impact on global trade and security, integrating academic research and data analytics.
29. Market Research Technology & Software Providers (Qualtrics, SurveyMonkey, Confirmit)
  - Produce a report on the latest innovations in survey technology and data analytics software for market research, including product comparisons, user case studies, and future trend forecasts.
  - Generate a comprehensive study on the integration of AI and machine learning in consumer insights platforms, highlighting case studies, performance metrics, and industry benchmarks.
  - Create a detailed report on digital transformation trends in market research technology, featuring analysis of leading software solutions, market share data, and recommendations for technology adoption.
When evaluating inputs, models tend to prefer the first response, prefer their own response, and prefer longer responses. ThursdAI
Real-time speech-to-text options for transcription:
- Deepgram has a MediaRecorder API, which is perfect.
- Whisper Streaming Web is a web app that can transcribe audio real-time from the browser. A good approach, but I wouldn’t use it for meeting transcription on my mid-end laptop. Streaming takes up the bulk of my GPU, leaving little for transcription.
- whisper-live runs as a Python console app and does something similar.
- Whisper WebGPU runs on the browser (only 200MB). Cool! But slow and still takes up GPU.
Mini-omni is an open-source Qwen-based LLM that can hear and talk while thinking in real-time. An interesting experiment, but not for prototyping.
OpenAI shares an insights report with clients that has insights on what different professions search for. What doctors search for is:
1. Is my diagnosis right?
2. How do I read this report?
3. Is my prescription correct?
4. Is there a cheaper medicine?
5. What’s the life expectancy given these symptoms?
Dataclasses in Python have a slight overhead over named tuples. The 2 main uses I see for them are: providing defaults and offering type hints.
UVB 76 is a radio channel has been broadcasting static (with occasional Russian conversation) since 1976. No one knows why. It’s live at https://m.youtube.com/watch?v=8h_D2P0iqMk
Romans washed clothes in urine. The government taxed the purchase of urine for commercial purposes! That’s the origin of the phrase “Pecunia non olet” which means “money doesn’t stink”.
Nix is a package manager that creates container-like environments. Like a cross between Docker and apt / venv. It has an immutable file system. DevBox is a higher-level tool built on top of Nix that streamlines developer workflows, e.g. common project environment setup.
VS Code can be used to develop inside a Docker container via Podman, too. Set dev.containers.dockerPath": "podman" Ref
Rill Data is an interesting BI tool based on DuckDB. It auto-generates a dashboard given a dataset.

It’s possible to assign “variables” in SQL (notably in DuckDB). Here’s an example:

WITH
  sessions AS (FROM events SELECT COUNT(DISTINCT session_id) AS value),
  pages AS (FROM events SELECT COUNT(*) AS value)
FROM sessions, pages
SELECT sessions.value / pages.value AS pages_per_session;

DuckDB has a GROUP BY * that groups by all categorical columns. SELECT x, y, COUNT(*) FROM t GROUP BY * is equivalent to SELECT x, y, COUNT(*) FROM t GROUP BY x, y.
VS Code can be used as a code executor by adding {"key": "shift+enter", "command": "workbench.action.terminal.runSelectedText", "when": "editorFocus"} to the keybindings.json file. Press Shift-Enter to run the selection on the terminal. Useful for DuckDB, SQLite, etc. Ref
LLMs are excellent at database migration. They can convert schemas and queries across SQL dialects (e.g. BigQuery to DuckDB, etc.) at 90%+ accuracy. This is useful when clients want to migrate cloud providers, go from on-prem to cloud, or reduce cost by switching databases.

Perl, 1994-2011

Sat, 28 Sep 2024 16:51:14 +0000

In 1994, I learnt Perl. It was fantastic. I used it to:

1995: Build CCChat - the unofficial IITM email system and software repository
1999: Build my entire blog from scratch
2000: Author my 2nd year thesis on the Behavioural Aspects of Financial Analysts by analyzing 600MB of IBES data
2002: Analyze where to place the central processing hubs for a bank
2004: Analyze the interest durations of public sector banks
2005: Creating music quizzes
2006: Create my own music search engine (which earned me about $100 a month in Google Ad revenue for a while)
2006: Automated resume filtering
2007: Create custom search engines
2008: Build application launchers

In 2006, I was convinced I should stick to Perl over Python.

In 2008, Google launched AppEngine and it provided free hosting (which was a big deal!) but had only 2 runtimes: Java and Python. The choice was clear. I’d rather learn Python than code in Java.

By 2011, I stopped installing Perl on my laptop.

Though most people know me mainly as a Python developer, I’ve programmed in Perl for about as long as I have in Python. I have fond memories of it. But I can’t read any of my code, nor write in it anymore.

When I watched The Perl Conference (now called The Perl and Raku Conference – Perl 6 is called Raku), I was surprised to hear how much the language had declined.

There were fewer than 100 attendees - and for 2025, they’ve decided to go smaller and book a tiny hotel, so as to break-even even if only 20 people show up.

Few languages have had as much of an impact on my life and thinking. My knowledge of modern programming comes from The Camel Book, functional programming from Higher Order Perl, Windows programming from Learning Perl on Win32 Systems, and so on. Even my philosophy of coding was shaped by Larry Wall’s the three great virtues of a programmer.

This is my homage to the language that shaped me. Bless you, Perl!

Things I Learned - 09 Jun 2024

Sun, 09 Jun 2024 00:00:00 +0000

This week, I learned:

httpretty can mock ALL Python HTTP libraries
Japanese pray to dead parents instead of gods. The dead are preserved in plates by priests. Japanese are generally non religious
Looks like GPT-4o is using CNNs to create vector embeddings of images, with images gridded into a 1x1, 2x2, etc. PLUS OCR. Ref
The sum of a sinusoidal series is like a spirogram. Spinning circle linked to another and so on https://www.andreinc.net/2024/04/24/from-the-circle-to-epicycles

Things I Learned - 18 Feb 2024

Sun, 18 Feb 2024 00:00:00 +0000

This week, I learned:

Fine tuning makes economic sense only if the input tokens SAVED is twice the output token size on each call.
Docker container memory usage on WSL2 docker stats
- frolvlad/alpine-glibc:alpine-3.17: 540KB
- ubuntu: 1MB (python3: +5MB)
- nikolaik/python-nodejs:python3.10-nodejs18-bullseye: 1.4MB (python3: +5MB)
- python:3-alpine: 612KB (python3: +7.5MB)
- python:3: 500KB (python3: +11.2MB)
- continuumio/miniconda3: 7.6MB (+6.5MB)
Discussion with Vinu Yamunan
- Databuck by FirstEigen. Autolysis plus monitoring
- Quality council has the data steward (maintainer of each dataset) coming together with the uses on a weekly basis to understand what quality problems to users are facing. Data owners jaundice at a lower frequency to get an understanding
- #TODO Automate rules for data quality in our projects and intranet
- Convert a config rule into business language. Explain SQL. These are good use cases for llm’s
- Graph DBs are powerful for flexible data structures, but query generation needs AI or expertise. Check the Neo4J language cypher
- Explore storing SAME data in relational DBs AND in graph DBs / document DBs for different use cases
- Dallas rocketry challenge. Build a rocket that can take an egg to 800 feet exactly and land without breaking it
Discussion with Karthik A
- #TODO Ask IIT students to do internship tasks. Use advent of code is a qualifying criterion
- Tata motors unionized DB admins for longevity. No one can take their jobs. Hires people who LIKE their jobs
- Rust gives me typing. It’s very efficient. Pola.rs is interesting but Pandas as good enough.
- Explore alerts from CCTV feeds. Karthik sends email alerts with pictures for:
  - “Is the machine on or off”? for productivity
  - “Are people not wearing helmets?” for safety at Cummins
- #TODO Integrate with WhatsApp. Use LLMs with function calling for responses
- Use expiring links (to pictures or content). It increases engagement
- Check Deno licensing. Is there a commercial clause? #ANS No - it’s MIT license
- Centre or excellence for zero emission tech at IIT. Karthik is part of it
- Explore auth0. 7000 users are free
- toml is part of the Python 3.11 standard library!
- If copilot writes code we don’t understand we are screwed. Hence expertise matters
Discussion with Vikas Kedia
- #TODO Plan an AMA
- The mind becomes lazy with financial success. Vikas is treating his podcast as a startup
- Hire a professional videographer for your content
- Financial RoI in financial markets is the highest. Programming is high too but FS is even better
- “Performative power” – when you’re forced to perform, you get better ideas
Observable 2.0 is an open source static site generator for data
Python dataclasses
SORA is OpenAI’s video generation model, and is stunning!
If Appa comes to Singapore even for a week, he will feel better and can boast to his friends. At over 90, it may be better to move Appa to where I am since many of his friends would be no more and shops, doctors, etc can be managed and getting an independent house nearby is not hard.
There is an SEZ in Gujarat where Indians can invest like in Mauritius without forex restraint
Shubha: Media sites are moving away from Vickrey auctions to first-price auctions for ads. That’s because they send the auction price forward to a search engine and the winning second-price value can lose even though the owner is willing to pay more. Second-price auctions don’t work unless ALL bidders are in the SAME auction. Ad networks are a hierarchy of auctions!
Gemini 1.5 launched.
Fly.io offers GPU hosting and auto stop when they have nothing to do.
Embeddings in random forest are very effective at classification – much better than dot product.
To deploy apps with OAuth + templating support in a small Docker container, use Caddy
Deno has native TypeScript, browser APIs, and compiles to multiple OSs
Ruff is a MUCH faster flake8
Two pass generation is a clever technique to get multiple SEQUENTIAL answers in a single API request. For example the schema {'code', 'optimized_code'} will generate code and then optimize it.
Unions in function calling allows flexible multi-step prompts in a single API.

Things I Learned - 14 Jan 2024

Sun, 14 Jan 2024 00:00:00 +0000

This week, I learned:

Transparent LED screens will be useful in windshieds to display maps as we drive.
Marimo is a reactive alternative to Jupyter notebooks that saves files as pure Python.
To run an org-specific chatbot on your own LLM: (via awesome-chatgpt)
- opengpts - but it doesn’t support auth
- chatbot-ui - but Supabase is hard to install
- anse - but it doesn’t support auth
- ChatGPT-Next-Web - but it doesn’t support auth
Python 3.13 gets a store and copy JIT
If an npm package adds another package as a dependency with version “*”, target package cannot unpublish ANY version! So this is a way of freezing EVERY repo and preventing unpublishing of EVERY version – an unintentional flaw in the npm design. via
Quantization is better than fewer parameters. So prefer high parameters (e.g. 70b) and quantize to 4-bit.
In-browser playgrounds has compiled WASM versions of Python, PHP, SQLite.
Happiness Lab podcast. Happiness lessons of the ancients
- Talking to strangers makes us happy
- Giving money makes us happy
- Free time makes us happier than working hard
Tangi
- Domain-specific models being beaten by general purpose models is a phase. It will reverse towards domain.
  - AI will potentially help build and understand domain-specific models
- Models are evolving so rapidly that humans cannot interpret models. We need a process to interpret models!
- xAI, Responsible AI, Physics-guided or Knowledge-guided models (called grey box models) are therefore a trend
- CS papers
  - Don’t review other papers, certainly not other fields.
  - Disregard measurement errors.
- When CS papers get applied to climate, manufacturing or biology, we’ll worry about
  - Interpretability
  - Domain-specific mechanics. (Introduce that into the training as a constraint.)
    - Many domain experts are using AI to UNDERSTAND their process. Need to explore
  - Uncertainty
- IB adds context to make learning applicable. But that distracts from the core learning, and if there’s a gap it widens
- Most data science courses teach “Python science”, not data science. They teach a bunch of models. They don’t teach how even one kind of model e.g. LSTM works.
- Most coaching programs today teach FAMILIARITY with problems, not critical thinking
- Most of current education will become redundant thanks to LLMs. For students AND teachers
  - Coding will become irrelevant
  - Cognitive thinking, reasoning, human relations, systems thinking will become more relevant
  - Troubleshooting will become more important. AI is not self-diagnosing. I would hire someone who can figure out something is going wrong, diagnose what’s going wrong, and fix it
  - #TODO Hire for troubleshooting ability. Give a Q, an A, and ask them to figure out if it’s wrong, why, and fix it
- All my exams and quizzes are open book, open ChatGPT. Onus is on me to give a problem that forces you to think.
  - #TODO Write a question paper that is ChatGPT proof.
- Exploring AI could be a ToK subject. “How to interact with an AI?”
  - We need a manual on how to use AI. Like Simon Willison says
  - Content doesn’t suffice. You need pedegogy. What to serve you at what time, how, how to assess. Lots of businesses are filling this gap
  - Students get great confidence when a teacher points to online content and says, I"ll tell you WHAT to see" and COMPLEMENTS that in their class
- “The map is not the territory.” Most people confuse sample mean for the actual.
- #ASK Parameter estimation -> Signal estimation -> State estimation
- Stats vs DL differ in that
  - There is no notion of a defined “truth”. Hence reliability is not measurable
  - Parameters have no value. Hence interpretability is ignored.
- #TODO Read 2020 National Education Policy. It’s quite modern.
  - We need a manual on self-learning too
  - Listening is not learning. You know only if you implement.
  - Levels for students:
    - I can solve it.
    - I can explain why it works.
    - I can find alternatives.
    - I can apply it to a new area, reformulating (requires imagination.)
  - For teachers, you also need:
    - Responsible learning (extra careful about what to teach and how to teach, to exceite them, to teach at THEIR level).
    - Show the universality and connecting to other concepts. E.g. noise reduction with FT is like using water to remove dirt. Transform to water domain, remove dirt, transform back to air domain. It’s better than dusting clothes to remove clothes. Washing machine programs are just different models of removing noise in the water domain.
- Teach people who WANT to learn AND who will APPLY it long-term. That’s what maximizes impact
  - Grad students are more satisfying that way.
  - Else, it is WASTED effort. (Not that it’s a bad thing for the student, but the effort IS wasted for the teacher)
- Therefore, I believe students should have general engineering first, and let students pick specialization later. Some universitie are doing that.
- #THINK Students remember my philosophy more than my content. We impart character, not just knowledge.
- Astrology and horoscopes serve a different function. They provide explainability, not predictive ability. As the world becomes less explainable, the need for astrology will grow.
  - Explainability is about creating STORIES that fits data plausibly. It has nothing to do with data or truth.
  - Explainability and predictive ability and reproducibility are all different. Maybe, Science is about the latter two, less about explainability.
  - Astrology is a model. The map is not the territory. It’s an explanatory, not a predictive model.
  - #THINK Therefore, my lessons are just explanations. Stats about experiments are STILL explanations. They are NOT reproducible or predictive. Hence not yet science
- The meaning of our life is the transformation we undergo in our lives
  - #TODO Read “The Journey of Souls” by Michael Newton. A hypnotherapist
- #TODO Try regression therapy / hypnosis. Record it and listen to it. Just for fun!
Rohini Deshpande
- Slam book was the Facebook of the 1900s
- Prepared mind is an extremely powerful tool for learning. Practice prepared mind
- When women drop out of education or career, that is also a waste from the teacher and system perspective
- The time for career growth is the same as child bearing time for women. That’s not true for men. But child rearing can be done by either. That’s not recognised. It’s 0K for a man to raise the child and make the home and 0K to treat that as the default
- Since men are more senior, it’s usually logical for them to stay in their jobs. That’s a systematic bias. When seniors advise women to step back. they respect it. That widens the barrier. Why not eliminate that situation?
- Be proud of the working women in the family
- Stats are just a symptom. They don’t explain the cause. (Map is not the territory.) Explanations are what really helps us fix the cause. Hence stories are important.
- Read Tinker Tailor Soldier Spy
RV Athimber health tips:
- Eat foods with low glycemic index
- Eliminate free salt completely
Voyage AI Embeddings have a higher quality, similar price compared to OpenAI embeddings. There’s a clear benefit to replacing text-embedding-3-large with voyage-3-lite. There’s a 200 MTok free tier currently.
mixtral-offloading cleverly loads only the model layer required at any point, letting you run Mixtral 8x7b on Colab Free and on 16GB GPUs. This notebook runs on Colab Free too.
CodeGPT is an alternative to Github Copilot that can use any LLM.

LLMs can teach experts

Thu, 02 Nov 2023 05:03:33 +0000

I am a fairly good programmer. So, when I see a problem, my natural tendency is to code.

I’m trying to break that pattern. Instead, I ask ChatGPT.

For example, I asked:

Write a compact 1-line Python expression that checks if user.id ends with @gramener.com or @straive.com

user.id.endswith(("@gramener.com", "@straive.com"))

After 15 years of using Python, I learnt that .endswith() supports tuple suffixes. This has been around since Python 2.5 (released in 2006 – before I knew Python.) The documentation has a tiny sentence in the middle saying “suffix can also be a tuple of suffixes to look for.”

I checked with a few colleagues, including Jaidev. They didn’t know it either.

It’s small little things like this that made me conclude.

I’m not going to code anymore. ChatGPT will, instead.

Maps, Delimitation, and Gerrymandering

Sun, 15 Aug 2021 06:04:36 +0000

I delivered a talk at PyCon India 2019. My slides are on Github.

This is a transcript of that talk.

What I’m going to be talking about is how you can get insights by joining two maps but before we go there, just some basic bookkeeping things.

In case you’re tweeting, these are the hashtags, you probably want to be using the #PyconIndia, my hashtag my IDs, #SANAND0, you don’t need to worry about the slides, they are online. I’ve already posted on Twitter, the link to the slide deck, the slide deck that you’re using but if you desperately do want to take notes, then one small suggestion. Research has shown that taking notes on pen and paper is much better than taking notes on laptops if you want to remember stuff or on mobile phones. So this was a discovery for me. In fact, it was my discovery of the year and I’m following it diligently. Do give it a shot if you want to take notes. Let’s dive in.

The story begins at the Karnataka elections in 2018. Say about one-eighth of the voters are Muslim, and both the Congress as well as the JD(S), were trying to get their support while on the other hand, BJP was taking potshots saying both of them are just trying to appease the community. The Hindu newspaper wanted to write a piece about how large a factor this is, and where all the Muslim vote is strong. You see, here we have a problem.

The thing is that the proportion of the population by religion is available only at the district level or the village level, depending on where you get the data from and this is from the census. Unfortunately, elections are not conducted by the district, elections are conducted by constituency and these are two very different maps. So, I have data in one map, which shows me how many Muslims exists in a particular region and I want to see how many Muslims live in a different region on another map and even though they overlap, there really is no direct way of getting the data from one layer on to the other. So, we literally don’t know how many Muslims live in a constituency.

So, how do we solve this problem? Well, the logical way is you could take one district and a constituency or a set of constituencies, and let’s say the district has a population of 100, out of which we know that 13% are Muslims and we want to split it evenly across a bunch of constituencies.

We could just overlay them. So one district could cover multiple constituencies, one constituency could cover multiple districts, and there is a many to many mapping between these there is sometimes full coverage, sometimes partial coverage.

So this district, for instance, covers at least one constituency fully and maybe this takes up about 1/3 of the total area. So I can say approximately 1/3 of the district’s population, which is that red area lives in this constituency.

Or let’s take another constituency that overlaps with this district. So, now only a portion of this constituency overlaps with this district. So in this area, which takes up maybe about 1/5, or about 20% of that district population, I can say that population lives here. In other words, we are simply making an assumption that within a district, which is the lowest level of data that you have, or if you have village data that’s far more granular, the population is uniformly distributed. That’s the basic assumption.

Now, what we can do is fragment each of these districts and constituencies by overlaying them and creating an intersection out of those, and reassembling those and this is a process that I call reshaping the map.

How much of this can we do in Python? There is a library called reshaper that we put together. The reshaper Library is something that’s very work in progress by the way. You can find it github.com/gramener/reshaper. It does exactly what I’m about to show you right now.

So let’s give it a shot. I’m going to open up the IPython notebook. The library that we are going to be using for this primarily, the core library is Geopandas. For those of you who have been working with data, Pandas is pretty much the de-facto library to use for any kind of data processing. Geopandas is becoming that kind of a standard for any shapefile. So if you have any shapefile and you want to do any kind of geospatial processing, an easy way of doing it is Geopandas and an easy way of installing it is through Conda using Anaconda. Rather than trying to do a pip install by yourself. It’s a little more efficient on most machines. So let’s import Geopandas. Now I have a shapefile that has the Karnataka census data which will eventually appear. I’m going to load it once it appears on the screen. (Just taking a long time. Okay, there we are back again.)

So GPD, which is the abbreviation for Geopandas, has a from_file function that lets you load any shapefile. Now, the other question you’ll have is where am I going to get these shapefiles from? We’ll come to that in a bit. It’s not as difficult as you might think. Let’s say you’ve downloaded the shapefiles. This particularly is the Karnataka census shapefile and what does this look like? Geopandas has a plotting function, which lets you see what the map looks like. So if you look at these districts, this is a pretty large district, this is Bangalore.

Let’s take the area for these. Geopandas offers an attribute called .geometry, which has an attribute called .area, which gets you the overall area of each of these regions, and if you want to look at what that data frame looks like, each of these regions corresponds to one row. So the Bagalkot district is one row, Bangalore rural district is one row and so on. All of the data in the shapefile also comes in here, you have a column called geometry, which has the additional geometry details, this is a pretty large column, which you probably won’t be going into the details of it. We’ve just now added one column called area, and this has the area of each of these regions and at the very least, you can figure out which are the larger regions, which are the smaller regions.

Let’s do the same for the constituencies data set. So here we have the constituencies that are more, these are parliamentary constituencies, by the way, not assembly constituencies. The difference being if you’re electing someone for the parliament, it’s or an MP, then it’s a parliamentary constituency. If you’re electing them for the assembly, which is an MLA, then it’s the assembly constituency. Parliamentary constituencies are bigger. So you’ll notice that out here, there are multiple parliamentary constituencies that sit in the same region that this district sits in, but it’s not a perfect match. Again, let’s take the area and see what this looks like. We have a bunch of these parliamentary constituencies like Gulbarga, Bijapur, etc, and their respective areas.

Now, Geopandas has a function called sjoin, which lets you take two shapefiles and create all the intersections around those shapefiles the fragments that I just showed you out here. So yeah, creating all these fragments is what the sjoin function does. So, if we do that, then what it’s now done is created a new data frame called merged and that has all these shapes. Let’s validate that. So there are 30 districts and 28 constituencies, but when you overlay them, it turns out that there are 147 fragments, each of which represents an intersection of a district and a constituency. Now, given this, it should be possible to just take any metric, like the percentage of Muslim voters or the number of Muslims, the size of the Muslim population, from the district data into the data that you have on the constituencies.

But it turns out that it’s a little trickier than that. So, you have to do a little more calculation and that’s what’s available in the reshaper library, you can take a look at the code, what it does is moves the metrics from one layer to another in a way that is seamless.

Once we have this, the result is an Excel sheet that kind of looks like this. It has all the attributes from both layers.

So, it says, for instance, that this particular assembly constituency is broken up into three regions, each of which maps to different districts. So some of it overlaps with more and some of it with shift saagar some of it with data and in fact, these are across different states and what is the area of each of these, along with a variety of other metrics that you can calculate and the proportion of area that is overlapping. Once you have this kind of data set, what can we do with it? So let’s revert back to our story.

What actually happened to the Muslim vote?

Well, this is the constituency-wise Muslim voter population in Karnataka.

This was used by the Hindu to publish an article around where exactly the bulk of the voters are concentrated. So, there is a chunk here, there’s a chunk here, there’s a chunk here.

Now, what was happening at this particular point was there was a fight for an alliance. The AIMIM, which is a Muslim party whose name is very long, and I can’t even say it fully. But they had won a number of seats in Telangana, and were looking to also participate in the Karnataka elections. They plan to contest in 60 seats. Now, to make sure that they get the Muslim vote, both JD(S) and the Congress were vying for an alliance with the party and in April 2018, AMIM decided that they will not be directly contesting in the elections, but instead would be supporting JD(S). Now, we have the results of the elections by constituency, we know the voter population by constituency. Let us see what happened to JD(S).

Turns out that where there were more Muslim populations, JD(S) actually got lower votes. So you can see the net result of this election and the alliance.

Congress, on the other hand, had a mildly higher vote share and where there was a significantly larger voter population.

It turns out that BJP was the one that gained the most.

Now while I’m moderately okay, at Python, I’m terrible at electoral analysis. So I have no idea what this means. Okay, I’ll let you figure it out. The elections in Maharashtra and Haryana are also coming up and it turns out that Congress is aligned with AIMIM and, well, let’s just leave it at that.

So, what can we do with this? What kinds of datasets exist and what is the potential of being able to join data sets across two spaces? That’s something that I’m pretty keen on.

It turns out that in India, there are broadly three kinds of geographic hierarchies. There is a political boundary hierarchy, a postal boundary hierarchy and an administrative boundary hierarchy.

By political boundary, I mean, the state parliamentary constituency, assembly constituency, going all the way down to polling booth. This has all of the results of the elections and one of the important aspects of this is that policies get made to a good extent at this level because the MPs and the MLS are focused on their respective constituencies.

The second is a postal code boundary. There is a zone within which there is a sorting district within which there is a post office and there is a PIN code, there are about 110,000 of these in total.

The third is the administrative boundary hierarchy. So there is a state there is a district, there’s also something called a division, but we’ll leave that aside, then there could be a sub district block or village, if it’s a rural area, or it could be municipality zone and ward if it’s a township.

Now, this apart, there is one other way we can create our own hierarchies. But before that, in case you’re looking for shapefiles, for many of these, the easiest way to get the shapefile for India is to search for “Datameet maps”. Datameet is a group that it’s a discussion forum and there is a lot of active discussion on various kinds of maps, pretty much any kind of map, there’s a decent chance that you’ll find it on Datameet and if it’s not there on Datameet, ask the people, they might be able to post something, and if not, it probably just doesn’t exist.

But you can also create your own boundaries. If you have a single location, you can look at the area that is closest to this particular location than any other location. So for example, if this were a network of, let’s say, schools, then what is that vision that is closest to a particular school than any other school.

So if I take this particular point as a school, then this red region represents all of those points which are closer to this school than any other school.

This particular process is called Voronoi tessellation and is something that comes out of box with QGIS, it’s something that you can create with the command line prompt again using the reshape or library, but what that means is that now you can take literally any point and convert that into a region and the potential for that is quite high.

So if I look at the kinds of data sets that you can create with location boundaries, right, so there’s… take all the hospitals, take all the schools, take all the bank branches, take all the petrol pumps take all the locations where crimes have been reported, take any address or take all the telephone towers, take all the locations where there are stores of a particular brand.

All of these are datasets for which you can get an address and an address can be geocoded into a point. If it can be geocoded into a point, you can convert that into a region and for each of these, you naturally have some kind of data for schools, you know how many teachers or how many students that are for telecom towers, you know, which is the organization that runs that tower, potentially the telecom organization will know how many calls are flowing through it, if it’s healthcare data, you know, how many facilities that hospital has, how many patients, how many doctors, all of these are data sets that can be added to that particular cell in your respective region.

But what this means, therefore, is that if we take any of these data sets that which you can create from location boundaries, or that often already exist by administrative boundaries – and this is a pretty powerful set as well. Census gives us demographic data, asset ownership, who owns laptops, internet connections, TV, cars, fridges, social and religious data, economic indicators, well, income, household indicators, is the house made of a mud roof brick roof, do you have a toilet in the house not have a toilet in the house, practically every government scheme is tracked this way. So how many people have benefited from the National Rural Employment Guarantee act? Banking data is reported this way health data is reported this way.

So effectively, anything that the government runs is reported by administrative boundaries. Anything that the corporate sector runs, by and large, is reported by locations. So between these two, there is enormous potential. But there’s also the fact of how decision making happens. Ultimately, political boundaries are owned, in some sense by an MP or an MLA. And, of course, there is also the associated IAS equivalents, who usually run it by administrative boundaries. So if I wanted somebody on the political side to make decisions, then I could take any of this data and put it on to the constituency boundaries. If I wanted an administrative official to make a decision, then I could take any of this data and put it on to a district. If I wanted a manager or a principal of a school, or the CEO of a hospital to make a decision, I could take all of that data and put it onto their geographic boundary.

For example, one of the things that the Hindu again did was found that the Congress is doing much better in the agrarian areas and they did that by taking the census data, which had the percentage of farmers and mapping that on to the voter constituency regions.

If we took, for example, census demographic data and school data, we can answer a question, where should we open new schools so that students don’t have to travel far or where there is a reasonably equal distribution of students across schools?

If we took economic indicators how well the country is growing versus bank branch data? Then we can answer questions like are the bank branches distributed? Based on population? That is, does every person roughly have equal access to the bank or based on wealth? Does every rupee have roughly equal access to the bank or if it’s in between? How close is it from one to the other?

We could find out whether increasing the district’s wealth leads to more theft. So that means people get richer. So does that mean that it does that lead to increase in crime? Or does it lead to less theft? Because the people are richer, and they don’t need to steal therefore? And these are data sets that are available and can be joined.

Similarly, with health data, does poor health lead to an increase in the number of pharmacies that are set up in that region because the pharmacies can sell more. Vice versa, if you actually set up more pharmacies? And does that have a positive impact on the people’s health in that particular region?

Now, the reason these questions are trivial to ask, but nearly impossible to solve today is because merging the data across different kinds of layers of maps is non-trivial. But both conceptually and technologically is quite an easy exercise.

What can we do to solve problems like this? Well, me personally, I’d love to see more of these hidden insights come out but there are a few things that you can do, literally right now.

First, if you have an idea, take a look at these data sets, any of the data sets that you know and raise an issue on this particular repository and I’d invite all of you to share this with people. It’ll be great to see what kinds of ideas can be solved using these problems and I’d like to crowd source this to a number of people on the administrative side, on the NGO side, and on the corporate side, to create a repository that says here are things that we can do.

If you want to try solving one of these and discovering your own insight, to build your own portfolio to share some useful knowledge. Then start by finding a map. Like I said, Datameet is a good place where you can find them up. You can find the reshaper library on https://github.com/gramener/reshaper. The links are again, on https://github.com/gramener/pycon2019. This is the one link that you need to remember and if you find something, do share it on Twitter. Please tag me @sanand0, I’d love to share it at least with the media and get some people to understand the power of geospatial joins.

If you want to contribute to the library right now it’s in a terrible state. Or if you want to learn more, I’m planning to organize a series of workshops on geospatial joins, do drop me an email. My email ID is s.anand@gramener.com and I’ll mail you the workshops.
If nothing else, if you just enjoyed the talk and you’ve learned something about it, then tweet about it. The tags are #PyconIndia2019, my ID, @sanand0. More than anything else, I’d love to see some insights come out by joining data.

Happy mapping!

Question: So my question is, basically, I’m belonging to northeast part of India. So I’m from Assam. So what happened in terms of the documentation for this geographic data and so those are always kept in a sort of, you know, register. We call registers or something, so how we use that image processing and all like, to enable those things into a more of a like a public space?

Answer: Okay. There are broadly three ways in which you can get this kind of data out. The first is beg, borrow, steal. Somebody in the government may have this data. So for example, if you go to the Survey of India, they sell these shape files. Of course, I’ve been trying to buy one of these shape files for the last six years now and have failed and I’ve tried it through the Prime Minister’s office and I still failed. But it’s actually easier to just walk over to the Survey of India office and give them a USB stick, and they’ll give it. So, depending on how you approach it, it may prove relatively straightforward.

On the other hand, sometimes the maps don’t exist. So for example, most interesting anecdote was the former head of the postal College of Mysore was trying to create a postal map, the region of all of the PIN codes. It turns out that nobody knows what the region is that a PIN code covers. So he created that he uploaded that ISRO’s Bhuvan, and then after about a year and a half realize that people have permission to upload into ISRO but download from Bhuvan. So after one and a half years of putting all the data, the data is locked, it’s not even there. So today, what is the best source of getting PIN code data? It turns out that what people did was took various locations, geocoded them, they said, this location is at that this particular PIN code, this location is at that this particular PIN code, let’s draw a region around it using the concept of Voronoi polygons, and publish it. So the second possibility is to create such maps.

The third possibility that you talked about, which is can we use image processing to detect it? Some features can be detected that way. So, for example, if you want to detect urban regions are constructed regions, that’s possible using satellite photography, if you want to locate water bodies, and whether they are growing or shrinking, so for example, in Chennai, the Chembarambakkam lake actually drying up, that’s something that you can draw a boundary around using image processing and that’s a straightforward method. But the thing is, I don’t think a single method will work for a wide variety of data sets, which is why we have many of these.

But the biggest lesson that I’ve learned is that 90% of the things that we want, somebody else has usually wanted, and has managed to get their inputs. So, I find that the most efficient ways to ask and Datameet is a pretty good place to ask if somebody already has his data.

Question: Anand, thank you very much for your talk. I’ve got a question regarding shape files. I had the requirement of using the map of India a few times and I suddenly realized that our external boundaries in a lot of places are in dispute and the kind of shape files that we get are not matching with what politically we want our file boundaries to be. So is there any official place from where we can get these shape files because the only shape file which are available are those distorted shape files, and I finally had to change the shape files myself to use it. I couldn’t find any official place from where to get the shape file.

Answer: So, the official place is the Survey of India, which claims to sell these maps like I said, for the last five, six years now I’ve been trying to buy these maps it’s actually not possible. But there are people who have succeeded and stock is being recorded, right? Okay. Let’s just say that if you go to Datameet maps, you will get unofficial but correct maps.

Question: So, since you are in the field, shouldn’t we have a system of getting correct official maps? Isn’t there a process being put in place or something?

Answer: I tried talking to a couple of people at the Prime Minister’s office and suggested this. They put me on the phone with the Inspector General of Surveys or some such high ranking official who said yes, absolutely, connected me to some person, who connected me to some person, who connected me to some person, who is exactly the same person I talked to in the first place. So, I don’t know. I’m sure there is a process. I don’t know it well enough.

Programming Minecraft with Websockets

Wed, 20 Jan 2021 03:40:09 +0000

Minecraft lets you connect to a websocket server when you’re in a game. The server can receive and send any commands. This lets you build a bot that you can … (well, I don’t know what it can do, let’s explore.)

Minecraft has commands you can type on a chat window. For example, type / to start a command and type setblock ~1 ~0 ~0 grass changes the block 1 north of you into grass. (~ means relative to you. Coordinates are specified as X, Y and Z.)

Note: These instructions were tested on Minecraft Bedrock 1.16. I haven’t tested them on the Java Edition.

Connect to Minecraft

You can send any command to Minecraft from a websocket server. Let’s use JavaScript for this.

First, run npm install ws uuid. (We need ws for websockets and uuid to generate unique IDs.)

Then create this mineserver1.js:

const WebSocket = require("ws");
const uuid = require("uuid"); // For later use

// Create a new websocket server on port 3000
console.log("Ready. On MineCraft chat, type /connect localhost:3000");
const wss = new WebSocket.Server({ port: 3000 });

// On Minecraft, when you type "/connect localhost:3000" it creates a connection
wss.on("connection", socket => {
  console.log("Connected");
});

On Minecraft > Settings > General > Profile, turn off the “Require Encrypted Websockets” setting.

Run node mineserver1.js. Then type /connect localhost:3000 in a Minecraft chat window. You’ll see 2 things:

MineCraft says “Connection established to server: ws://localhost:3000”
Node prints “Connected”

Now, our program is connected to Minecraft, and can send/receive messages.

Notes:

The Python equivalent is in mineserver1.py. Run python mineserver1.py.
If you get an Uncaught Error: Cannot find module 'ws', make sure you ran npm install ws uuid.
If you get an “Encrypted Session Required” error, make sure you turned off the “Require Encrypted Websockets” setting mentioned above.
To disconnect, run /connect off

Now let’s listen to the players’ chat.

A connected websocket server can send a “subscribe” message to Minecraft saying it wants to “listen” to specific actions. For example, you can subscribe to “PlayerMessage”. Whenever a player sents a chat message, Minecraft will notify the websocket client.

Here’s how to do that. Add this code in the wss.on('connection', socket => { ... }) function.

// Tell Minecraft to send all chat messages. Required once after Minecraft starts
socket.send(JSON.stringify({
  "header": {
    "version": 1, // We're using the version 1 message protocol
    "requestId": uuid.v4(), // A unique ID for the request
    "messageType": "commandRequest", // This is a request ...
    "messagePurpose": "subscribe", // ... to subscribe to ...
  },
  "body": {
    "eventName": "PlayerMessage", // ... all player messages.
  },
}));

Now, every time a player types something in the chat window, the socket will receive it. Add this code below the above code:

// When MineCraft sends a message (e.g. on player chat), print it.
socket.on("message", packet => {
  const msg = JSON.parse(packet);
  console.log(msg);
});

This code parses all the messages it receives and prints them.

This code in is mineserver2.js. Run node mineserver2.js. Then type /connect localhost:3000 in a Minecraft chat window. Then type a message (e.g. “alpha”) in the chat window. You’ll see a message like this in the console.

{
  header: {
    messagePurpose: 'event',        // This is an event
    requestId: '00000000-0000-0000-0000-000000000000',
    version: 1                      // using version 1 message protocol
  },
  body: {
    eventName: 'PlayerMessage',
    measurements: null,
    properties: {
      AccountType: 1,
      ActiveSessionID: 'e0afde71-9a15-401b-ba38-82c64a94048d',
      AppSessionID: 'b2f5dddc-2a2d-4ec1-bf7b-578038967f9a',
      Biome: 1,                     // Plains Biome. https://minecraft.gamepedia.com/Biome
      Build: '1.16.201',            // That's my build
      BuildNum: '5131175',
      BuildPlat: 7,
      Cheevos: false,
      ClientId: 'fcaa9859-0921-348e-bc7c-1c91b72ccec1',
      CurrentNumDevices: 1,
      DeviceSessionId: 'b2f5dddc-2a2d-4ec1-bf7b-578038967f9a',
      Difficulty: 'NORMAL',         // I'm playing on normal difficulty
      Dim: 0,
      GlobalMultiplayerCorrelationId: '91967b8c-01c6-4708-8a31-f111ddaa8174',
      Message: 'alpha',             // This is the message I typed
      MessageType: 'chat',          // It's of type chat
      Mode: 1,
      NetworkType: 0,
      Plat: 'Win 10.0.19041.1',
      PlayerGameMode: 1,            // Creative. https://minecraft.gamepedia.com/Commands/gamemode
      Sender: 'Anand',              // That's me.
      Seq: 497,
      WorldFeature: 0,
      WorldSessionId: '8c9b4d3b-7118-4324-ba32-c357c709d682',
      editionType: 'win10',
      isTrial: 0,
      locale: 'en_IN',
      vrMode: false
    }
  }
}

Notes:

The Python equivalent is in mineserver2.py. Run python mineserver2.py.
The official Minecraft docs say that the MCWSS protocol is outdated. But it seems to work.
The full list of things we can subscribe to is undocumented, but @jocopa3 has reverse-engineered a list of messages we can subscribe to, and they’re somewhat meaningful.
This Go package has code that explores the protocol further.
This chat has more details. There’s also an outdated list of JSON messages from @jocopa3.
Here’s a sample program that places a block in Minecraft

Build structures using chat

Let’s create a pyramid of size 10 around us when we type pyramid 10 in the chat window.

The first step is to check if the player sent a chat message like pyramid 10 (or another number). Add this code below the above code:

// When MineCraft sends a message (e.g. on player chat), act on it.
socket.on("message", packet => {
  const msg = JSON.parse(packet);
  // If this is a chat window
  if (msg.body.eventName === "PlayerMessage") {
    // ... and it's like "pyramid 10" (or some number), draw a pyramid
    const match = msg.body.properties.Message.match(/^pyramid (\d+)/i);
    if (match) {
      draw_pyramid(+match[1]);
    }
  }
});

If the user types “pyramid 3” on the chat window, draw_pyramid(3) is called.

In draw_pyramid(), let’s send commands to build a pyramid. To send a command, we need to create a JSON with the command (e.g. setblock ~1 ~0 ~0 grass). Add this code below the above code:

function send(cmd) {
  const msg = {
    "header": {
      "version": 1,
      "requestId": uuid.v4(), // Send unique ID each time
      "messagePurpose": "commandRequest",
      "messageType": "commandRequest",
    },
    "body": {
      "version": 1, // TODO: Needed?
      "commandLine": cmd, // Define the command
      "origin": {
        "type": "player", // Message comes from player
      },
    },
  };
  socket.send(JSON.stringify(msg)); // Send the JSON string
}

Let’s write draw_pyramid() to create a pyramid using glowstone by adding this code below the above code:

// Draw a pyramid of size "size" around the player.
function draw_pyramid(size) {
  // y is the height of the pyramid. Start with y=0, and keep building up
  for (let y = 0; y < size + 1; y++) {
    // At the specified y, place blocks in a rectangle of size "side"
    let side = size - y;
    for (let x = -side; x < side + 1; x++) {
      send(`setblock ~${x} ~${y} ~${-side} glowstone`);
      send(`setblock ~${x} ~${y} ~${+side} glowstone`);
      send(`setblock ~${-side} ~${y} ~${x} glowstone`);
      send(`setblock ~${+side} ~${y} ~${x} glowstone`);
    }
  }
}

This code in is mineserver3.js.

Run node mineserver3.js.
Then type /connect localhost:3000 in a Minecraft chat window.
Then type pyramid 3 in the chat window.
You’ll be surrounded by a glowstone pyramid.

Notes:

The Python equivalent is in mineserver3.py. Run python mineserver3.py.
The “requestId” needs to be a UUID – at least for block commands. I tried unique “requestId” values like 1, 2, 3 etc. That didn’t work.

Understand Minecraft’s responses

For every command you send, Minecraft sends a response. It’s “header” looks like this:

{
  "header": {
    "version": 1,
    "messagePurpose": "commandResponse",                  // Response to your command
    "requestId": "97dee9a3-a716-4caa-aef9-ddbd642f2650"   // ... and your requestId
  }
}

If the command is successful, the response has body.statusCode == 0. For example:

{
  "body": {
    "statusCode": 0,                  // No error
    "statusMessage": "Block placed",  // It placed the block you wanted
    "position": { "x": 0, "y": 64, "z": 0 }   // ... at this location
  },
}

If the command failed, the response has a negative body.statusCode. For example:

{
  "body": {
    "statusCode": -2147352576,        // This is an error
    "statusMessage": "The block couldn't be placed"
  },
}

To print these, add this to socket.on('message', ...):

// If we get a command response, print it
if (msg.header.messagePurpose == "commandResponse") {
  console.log(msg);
}

This code in is mineserver4.js.

Run node mineserver4.js.
Then type /connect localhost:3000 in a Minecraft chat window.
Then type pyramid 3 in the chat window.
You’ll be surrounded by a glowstone pyramid, and the console will show every command response.

Notes on common error messages:

The block couldn't be placed (-2147352576): The same block was already at that location.
Syntax error: Unexpected "xxx": at "~0 ~9 ~-1 >>xxx<<" (-2147483648): You gave wrong arguments to the command.
Too many commands have been requested, wait for one to be done (-2147418109): Minecraft only allows 100 commands can be executed without waiting for their response.
More error messages here.

Wait for commands to be done

Typing “pyramid 3” works just fine. But try “pyramid 5” and your pyramid is incomplete.

That’s because Minecraft only allows up to 100 messages in its queue. On the 101st message, you get a Too many commands have been requested, wait for one to be done error.

{
  "header": {
    "version": 1,
    "messagePurpose": "error",
    "requestId": "a5051664-e9f4-4f9f-96b8-a56b5783117b"
  },
  "body": {
    "statusCode": -2147418109,
    "statusMessage": "Too many commands have been requested, wait for one to be done"
  }
}

So let’s modify send() to add to a queue and send in batches. We’ll create two queues:

const sendQueue = []; // Queue of commands to be sent
const awaitedQueue = {}; // Queue of responses awaited from Minecraft

In wss.on('connection', ...), when Minecraft completes a command, we’ll remove it from the awaitedQueue. If the command has an error, we’ll report it.

// If we get a command response
if (msg.header.messagePurpose == "commandResponse") {
  // ... and it's for an awaited command
  if (msg.header.requestId in awaitedQueue) {
    // Print errors 5(if any)
    if (msg.body.statusCode < 0) {
      console.log(
        awaitedQueue[msg.header.requestId].body.commandLine,
        msg.body.statusMessage,
      );
    }
    // ... and delete it from the awaited queue
    delete awaitedQueue[msg.header.requestId];
  }
}
// Now, we've cleared all completed commands from the awaitedQueue.

Once we’ve processed Minecraft’s response, we’ll send pending messages from sendQueue, upto 100 and add them to the awaitedQueue.

// We can send new commands from the sendQueue -- up to a maximum of 100.
let count = Math.min(100 - Object.keys(awaitedQueue).length, sendQueue.length);
for (let i = 0; i < count; i++) {
  // Each time, send the first command in sendQueue, and add it to the awaitedQueue
  let command = sendQueue.shift();
  socket.send(JSON.stringify(command));
  awaitedQueue[command.header.requestId] = command;
}
// Now we've sent as many commands as we can. Wait till the next PlayerMessage/commandResponse

Finally, in function send(), instead of socket.send(JSON.stringify(msg)), we use sendQueue.push(msg) to add the message to the queue.

This code in is mineserver5.js.

Run node mineserver5.js.
Then type /connect localhost:3000 in a Minecraft chat window.
Then type pyramid 6 in the chat window.
You’ll be surrounded by a large glowstone pyramid.
The console will print messages like setblock ~0 ~6 ~0 glowstone The block couldn't be placed because we’re trying to place duplicate blocks.

Comments

Rodland 8 Mar 2021 12:45 am:Thanks for a great tutorial.However when I try to implement the chat listener (server2.js) i get an JSON error from MineCraft (in node.js console):“kryptert økt kreves” = “Needs secure connection”
Ready. On MineCraft chat, type /connect localhost:3000 Connected { body: { statusCode: -2147418107, statusMessage: ‘Kryptert økt kreves’ }, header: { messagePurpose: ’error’, requestId: ‘3522bb53-5256-4030-b827-4b411ae72a6c’, version: 1 } }
S Anand 8 Mar 2021 4:07 pm: Under Settings > General > Profile, please turn off the “Require Encrypted Websockets” setting. I’ve added this to the tutorial. Thanks for flagging this!
S Anand 2 Jan 2022 9:48 am: That’s right. Looks like you resolved it and changed localhost to minecraft-server.warpedwartwars.repl.co 👍
WarpedWartWars 21 Jun 2022 6:48 am: Got it working on localhost, using the Python mineserver3.py, but the message protocol seems to have changed, to this: {‘body’: {‘message’: ‘pyramid 32’, ‘receiver’: ‘’, ‘sender’: ‘WarpedWartWars’, ’type’: ‘chat’}, ‘header’: {’eventName’: ‘PlayerMessage’, ‘messagePurpose’: ’event’, ‘version’: 16842752}}
WarpedWartWars 15 Dec 2021 8:27 am: I tried this, as it seemed to be the best guide I could find. I made a Replit Repl: https://replit.com/@WarpedWartWars/minecraft-server?v=1 and I ran it, then tried “/connect localhost:3000” in Minecraft Bedrock 1.18.2, and I got “Could not connect to server: ws://localhost:3000”. I imagine I have to change “localhost” to something else.
someGuy 23 Feb 2022 8:11 am: This will be deleted in couple of days but some of the reasons people are getting issues connecting is because of UWP loopback restrictions source: https://stackoverflow.com/questions/34589522/cant-see-localhost-from-uwp-app to resolve this problem for windows: https://docs.microsoft.com/en-us/minecraft/creator/documents/scriptdevelopertools. run this in system administrator command prompt: CheckNetIsolation.exe LoopbackExempt -a -p=S-1-15-2-1958404141-86561845-1752920682-3514627264-368642714-62675701-733520436

How to extend Markdown with custom blocks

Fri, 08 Jan 2021 09:37:48 +0000

One problem I’ve had in Markdown is rendering a content in columns.

On Bootstrap, the markup would look like this:

<div class="row">
  <div class="col">...</div>
  <div class="col">...</div>
</div>

How do we get that into Markdown without writing HTML?

On Python, the attribute lists extension lets you add a class. For example:

This is some content
{: .row}

… renders <p class="row">This is some content</p>.

But I can’t do that to multiple paragraphs. Nor can I next content, i.e. add a .col inside the .row.

Enter markdown-customblocks. It’s a Python module that extends Python Markdown. This lets me write:

::: row
::: col
Content in column 1
::: col
Content in column 2
::: row
::: col
Content in column 1
::: col
Content in column 2

This translates to:

<div class="row">
  <div class="col">Content in column 1</div>
  <div class="col">Content in column 2</div>
</div>

Better yet, we can create our own custom HTML block types. For example, this code:

from markdown import Markdown
from customblocks import CustomBlocksExtension

def audio(ctx, src, type="audio/mp3"):
    return f'''<audio src="{src}" type="{type}">
      {ctx.content}
    </audio>'''

md = Markdown(extensions=[
    CustomBlocksExtension(generators={
        'audio': audio
    }),
])

… lets you convert this piece of Markdown:

md.convert("""
::: audio src="mymusic.ogg" type="audio/ogg"
    Your browser does not support `audio`.
""")

… into this HTML:

<audio src="mymusic.ogg" type="audio/ogg">
  Your browser does not support <code>audio</code>.
</audio>

markdown-customblocks is easily the most useful Python module I discovered last quarter.

lxml is fast enough

Fri, 11 Dec 2020 03:12:34 +0000

Given the blazing speed of Node.js these days, I expected HTML parsing to be faster on Node than on Python.

So I compared lxml with htmlparser2 – the fastest libraries on Python and JS in parsing the reddit home page (~700KB).

lxml took ~8.6 milliseconds
htmlparser2 took ~14.5 milliseconds

Looks like lxml is much faster. I’m likely to stick around with Python for pure HTML parsing (without JavaScript) for a while longer.

In [1]: from lxml.html import parse

In [2]: %timeit tree = parse('reddit.html')
8.69 ms ± 190 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

const { Parser } = require("htmlparser2");
const { DomHandler } = require("domhandler");
const fs = require("fs");
const html = fs.readFileSync("reddit.html");
const handler = new DomHandler(function(error, dom) {});

const start = +new Date();
for (var i = 0; i < 100; i++) {
  const parser = new Parser();
  parser.write(html);
  parser.end();
}
const end = +new Date();
console.log((end - start) / 100);

Note: If I run the htmlparser2 code 100 times instead of 10, it only takes 7ms per loop. The more the number of loops, the faster it parses. I guess Node.js optimizes repeated loops. But I’m only interested in the first iteration, since I’ll be parsing files only once.

The most popular scientific Python modules

Tue, 11 Dec 2012 02:37:52 +0000

I just scraped the scientific packages on pypi. Here are the top 50 by downloads.

Name	Description	Size	Downloads
numpy	NumPy: array processing for numbers, strings, records, and objects.	2000000	133076
scipy	SciPy: Scientific Library for Python	7000000	33990
pygraphviz	Python interface to Graphviz	99000	22828
geopy	Python Geocoding Toolbox	32000	18617
googlemaps	Easy geocoding, reverse geocoding, driving directions, and local search in Python via Google.	69000	15135
Rtree	R-Tree spatial index for Python GIS	495000	14370
nltk	Natural Language Toolkit	1000000	12844
Shapely	Geometric objects, predicates, and operations	93000	12635
pyutilib.component.doc	Documentation for the PyUtilib Component Architecture.	372000	10181
geojson	Encoder/decoder for simple GIS features	12000	9407
GDAL	GDAL: Geospatial Data Abstraction Library	410000	8957
scikits.audiolab	A python module to make noise from numpy arrays	1000000	8856
pupynere	NetCDF file reader and writer.	16000	8809
scikits.statsmodels	Statistical computations and models for use with SciPy	3000000	8761
munkres	munkres algorithm for the Assignment Problem	42000	8409
scikit-learn	A set of python modules for machine learning and data mining	2000000	7735
networkx	Python package for creating and manipulating graphs and networks	1009000	7652
pyephem	Scientific-grade astronomy routines	927000	7644
PyBrain	PyBrain is the swiss army knife for neural networking.	255000	7313
scikits.learn	A set of python modules for machine learning and data mining	1000000	7088
obspy.seisan	SEISAN read support for ObsPy.	3000000	6990
obspy.wav	WAV(audio) read and write support for ObsPy.	241000	6985
obspy.seishub	SeisHub database client for ObsPy.	237000	6941
obspy.sh	Q and ASC (Seismic Handler) read and write support for ObsPy.	285000	6926
crcmod	CRC Generator	128000	6714
obspy.fissures	DHI/Fissures request client for ObsPy.	1000000	6339
stsci.distutils	distutils/packaging-related utilities used by some of STScI's packages	25000	6215
pyopencl	Python wrapper for OpenCL	1000000	6124
Kivy	A software library for rapid development of hardware-accelerated multitouch applications.	11000000	5879
speech	A clean interface to Windows speech recognition and text-to-speech capabilities.	17000	5809
patsy	A Python package for describing statistical models and for building design matrices.	276000	5517
periodictable	Extensible periodic table of the elements	775000	5498
pymorphy	Morphological analyzer (POS tagger + inflection engine) for Russian and English (+perhaps German) languages.	70000	5174
imposm.parser	Fast and easy OpenStreetMap XML/PBF parser.	31000	4940
hcluster	A hierarchical clustering package for Scipy.	442000	4761
obspy.core	ObsPy - a Python framework for seismological observatories.	487000	4608
Pyevolve	A complete python genetic algorithm framework	99000	4509
scikits.ann	Approximate Nearest Neighbor library wrapper for Numpy	82000	4368
obspy.imaging	Plotting routines for ObsPy.	324000	4356
obspy.xseed	Dataless SEED, RESP and XML-SEED read and write support for ObsPy.	2000000	4331
obspy.sac	SAC read and write support for ObsPy.	306000	4319
obspy.arclink	ArcLink/WebDC client for ObsPy.	247000	4164
obspy.iris	IRIS Web service client for ObsPy.	261000	4153
Orange	Machine learning and interactive data mining toolbox.	14000000	4099
obspy.neries	NERIES Web service client for ObsPy.	239000	4066
pandas	Powerful data structures for data analysis, time series,and statistics	2000000	4037
pycuda	Python wrapper for Nvidia CUDA	1000000	4030
GeoAlchemy	Using SQLAlchemy with Spatial Databases	159000	3881
pyfits	Reads FITS images and tables into numpy arrays and manipulates FITS headers	748000	3746
HTSeq	A framework to process and analyze data from high-throughput sequencing (HTS) assays	523000	3720
pyopencv	PyOpenCV - A Python wrapper for OpenCV 2.x using Boost.Python and NumPy	354000	3660
thredds	THREDDS catalog generator.	25000	3622
hachoir-subfile	Find subfile in any binary stream	16000	3540
fluid	Procedures to study geophysical fluids on Python.	210000	3520
pygeocoder	Python interface for Google Geocoding API V3. Can be used to easily geocode, reverse geocode, validate and format addresses.	7000	3514
csc-pysparse	A fast sparse matrix library for Python (Commonsense Computing version)	111000	3455
topex	A very simple library to interpret and load TOPEX/JASON altimetry data	7000	3378
arrayterator	Buffered iterator for big arrays.	7000	3320
python-igraph	High performance graph data structures and algorithms	3000000	3260
csvkit	A library of utilities for working with CSV, the king of tabular file formats.	29000	3236
PyVISA	Python VISA bindings for GPIB, RS232, and USB instruments	237000	3201
Quadtree	Quadtree spatial index for Python GIS	40000	3000
ProxyHTTPServer	ProxyHTTPServer -- from the creator of PyWebRun	3000	2991
mpmath	Python library for arbitrary-precision floating-point arithmetic	1000000	2901
bigfloat	Arbitrary precision correctly-rounded floating point arithmetic, via MPFR.	126000	2879
SimPy	Event discrete, process based simulation for Python.	5000000	2871
Delny	Delaunay triangulation	18000	2790
pymc	Markov Chain Monte Carlo sampling toolkit.	1000000	2727
PyBUFR	Pure Python library to encode and decode BUFR.	10000	2676
collective.geo.bundle	Plone Maps (collective.geo)	11000	2676
dap	DAP (Data Access Protocol) client and server for Python.	125000	2598
rq	RQ is a simple, lightweight, library for creating background jobs, and processing them.	29000	2590
pyinterval	Interval arithmetic in Python	397000	2558
StarCluster	StarCluster is a utility for creating and managing computing clusters hosted on Amazon's Elastic Compute Cloud (EC2).	2000000	2521
fisher	Fast Fisher's Exact Test	43000	2503
mathdom	MathDOM - Content MathML in Python	169000	2482
img2txt	superseded by asciiporn, http://pypi.python.org/pypi/asciiporn	443000	2436
DendroPy	A Python library for phylogenetics and phylogenetic computing: reading, writing, simulation, processing and manipulation of phylogenetic trees (phylogenies) and characters.	6000000	2349
geolocator	geolocator library: locate places and calculate distances between them	26000	2342
MyProxyClient	MyProxy Client	67000	2325
PyUblas	Seamless Numpy-UBlas interoperability	51000	2252
oroboros	Astrology software	1000000	2228
textmining	Python Text Mining Utilities	1000000	2198
scikits.talkbox	Talkbox, a set of python modules for speech/signal processing	147000	2188
asciitable	Extensible ASCII table reader and writer	312000	2160
scikits.samplerate	A python module for high quality audio resampling	368000	2151
tabular	Tabular data container and associated convenience routines in Python	52000	2114
pywcs	Python wrappers to WCSLIB	2000000	2081
DeliciousAPI	Unofficial Python API for retrieving data from Delicious.com	19000	2038
hachoir-regex	Manipulation of regular expressions (regex)	31000	2031
Kamaelia	Kamaelia - Multimedia & Server Development Kit	2000000	2007
seawater	Seawater Libray for Python	2000000	1985
descartes	Use geometric objects as matplotlib paths and patches	3000	1983
vectorformats	geographic data serialization/deserialization library	10000	1949
PyMT	A framework for making accelerated multitouch UI	18000000	1945
times	Times is a small, minimalistic, Python library for dealing with time conversions between universal time and arbitrary timezones.	4000	1929
CocoPy	Python implementation of the famous CoCo/R LL(k) compiler generator.	302000	1913
django-shapes	Upload and export shapefiles using GeoDjango.	9000	1901
sympy	Computer algebra system (CAS) in Python	5000000	1842
pyfasta	fast, memory-efficient, pythonic (and command-line) access to fasta sequence files	14000	1836

Comments

Ron Z 25 Jun 2013 10:19 pm: Another good one is simpleCV: http://www.simplecv.org/
Ravindranath M 8 Apr 2013 7:13 am: Nice list Anand. Very useful. – This is Ravi from Comviva, who attended your introduction.

Inspecting code in Python

Tue, 04 Sep 2012 03:49:08 +0000

Lisp users would laugh, since they have macros, but Python supports some basic code inspection and modification.

Consider the following pieces of code:

margin = lambda v: 1 - v['cost'] / v['sales']

What if you wanted another function that lists all the dictionary indices used in the function? That is, you wanted to extract cost and sales?

This is a real-life problem I encountered this morning. I have 100 functions, each defining a metric. For example,

lambda v: v['X'] + v['Y']
lambda v: v['X'] - v['Z']
lambda v: (v['X'] + v['Y']) / v['Z']
…

I had to plot the functions, as well as each of the corresponding elements (‘X’, ‘Y’ and ‘Z’) in the formula.

Two options. One: along with each formula, maintain a list of the elements used. Two: figure it out automatically.

Each function has a func_code attribute. So, when you take

margin = lambda v: 1 - v['cost'] / v['sales']

margin.func_code is a “code object”. This has a bunch of interesting attributes, one of which is co_consts

>>> margin.func_code.co_consts
(None, 1, 'cost', 'sales')

There – I just pick the strings out of that list and we’re done (for simple functions at least.)

Check out http://docs.python.org/reference/datamodel.html and search for func_ – you’ll find a number of interesting things you can do with functions, such as

Finding and changing the default parameters
Accessing the global variables of the namespace where the function was defined (!)
Replacing the function code with new code

Also search for co_ – you’ll find some even more interesting things you can do with the code:

Find all local variable names
Find all constants used in the code
Find the filename and line number where the code was compiled from

Python also comes with a disassembly module dis. A look at its source is instructive.

Comments

Kamaal 4 Sep 2012 7:29 am: This is cool. But Python needs to get Multiline Lambdas. One of the things Higher order Perl programmers miss in Python are muliline lamdas.
S Anand 4 Sep 2012 11:40 am: Yeah – I miss that too. But I guess it’d go too strongly against the Python ethos of readability. I find it interesting that I miss multi-line lambdas ONLY when writing code. Never when reading it. So guess it isn’t too bad a thing, given that I’m reading (my own, often) code 90% of the time, and writing it less than 10% :-)

python on S Anand

Proving Code Works with Z3

Z3 can find counterexamples

ChatGPT can run Z3 to find issues

Some issues are not bad

But not all issues are worth reporting

Still, Z3 + ChatGPT is quite effective

Sambar Styles

Things I Learned - 18 Jan 2026

Finding open source bugs with Ty

Get this blog via email on Google Groups

Things I Learned - 29 Jun 2025

Things I Learned - 27 Apr 2025

Things I Learned - 02 Mar 2025

Perl, 1994-2011

Things I Learned - 09 Jun 2024

Things I Learned - 18 Feb 2024

Things I Learned - 14 Jan 2024

LLMs can teach experts

Maps, Delimitation, and Gerrymandering

Programming Minecraft with Websockets

Connect to Minecraft

Subscribe to chat messages

Build structures using chat

Understand Minecraft’s responses

Wait for commands to be done

Comments

Rodland 8 Mar 2021 12:45 am:Thanks for a great tutorial.However when I try to implement the chat listener (server2.js) i get an JSON error from MineCraft (in node.js console):“kryptert økt kreves” = “Needs secure connection”

How to extend Markdown with custom blocks

lxml is fast enough

The most popular scientific Python modules

Comments

Inspecting code in Python

Comments