parquet on S Anand

Things I Learned - 31 May 2026

Sun, 31 May 2026 00:00:00 +0000

This week, I learned:

D-ID is an avatar generator platform like HeyGen. Creatify and Synthesia are a couple of others I heard of. This space seems to be growing.
cosign is a CLI that lets you sign and verify any piece of text with a Google, GitHub or Microsoft account. cosign sign-blob FILE --bundle sign.json opens a login window and creates a sign.json signature. Anyone who has FILE and sign.json and the email ID can verify via a Google account with cosign verify-blob FILE --bundle sign.json --certificate-identity $EMAIL --certificate-oidc-issuer https://accounts.google.com.
arxiv2md.org converts arXiv papers to Markdown. Source. markxiv.org claims the same - by just changing the URL - but it ended up reporting an error when I tried this link: https://markxiv.org/abs/2604.08649.
From Akhilesh Tilotia: So we have someone in our team with initials AS. She made a document which was named vAS. Then I made edits and named it vAT. These docs were in a CoWork folder. I asked Claude to clean up my doc. It created another version for me to review. In its wisdom, it named the file vAU 🙂
Maybe what a forward-deployed engineer does is enginer AI-native workflows. (This sounded profound when I wrote it down. Not sure if it’ll sound as profound tomorrow.) The idea is that the FDE will say, screw existing processes; let me fire up my AI agent and get stuff done; THEN we’ll figure out what works, how to optimize it, etc.
The PRAGMA: Revolut Foundation Model has some good tokenization ideas for tabular data. Create your own token space with key–value–time tokenization - to retain field information. Bucketize numbers by percentile, preserving magnitude/ordering that subword tokenization destroys. Encode time both as log-seconds and as cyclical calendar features.
Codex uses the Alt + Up Arrow key to edit queued commands, but on the VS Code terminal, this key binding is not sent to the terminal. Enable the terminal.integrated.sendKeybindingsToShell setting to send it to the terminal, hence Codex.
Based on this catalog on “universal foods”, here’s what I 🟢 like, am 🟡 neutral, 🔴 dislike, 🟣 must try, and will ⚫ skip.
- Universal favorites: 🟢 pizza, 🟢 fried potatoes/chicken, 🟡 dumplings, 🟢 ice cream.
- Universal comfort foods: 🟢 khichdi, 🟡 congee, 🟡 dal-rice, 🟡 risotto, 🟡 ramen, 🟢 pho, ⚫ chicken noodle soup, 🔴 rice porridge, 🟡 mac-and-cheese, 🔴 mashed potato, 🟣 polenta, 🟢 oatmeal, 🟣 Japanese curry rice.
- Acquired tastes that convert most: 🟡 coffee, 🟢 tea, 🟡 dark chocolate, 🟢 mild fermented dairy, 🟢 pickles, 🟢 olives, 🟣 kimchi, 🟣 miso, 🟢 mild chili dishes.
- Acquired tastes that have cult devotion: 🟣 durian, 🟣 natto, 🟣 stinky tofu, ⚫ fermented fish, ⚫ hákarl, 🟢 very funky blue cheese, ⚫ offal.
OceanoPDF seems like a good place to download ePubs of books.
The entire Wikipedia is available as a Parquet file. You can query it like duckdb -c "FROM 'hf://datasets/wikimedia/structured-wikipedia/enwiki/data/*.parquet' LIMIT 5". The English version has 35 GB, 7.6 million articles, and you’re better off downloading it rather than running analyses remotely.
When you receive a Calendly link of the form https://cal.com/USER/EVENT you can fetch the available slots via curl -H 'cal-api-version: 2024-09-04' 'https://api.cal.com/v2/slots?eventTypeSlug=EVENT&username=USER&start=2026-05-25&end=2026-06-01&timeZone=Asia/Singapore&format=range'. Useful to automate good meeting-slot selection.
“Reference saved memories” in ChatGPT is different from “Reference chat history” as per OpenAI. In Developer Mode, memory is turned off, but not chat history. I confirmed that I can access past conversations in Developer Mode. It might be a privacy concern for others, but for me, this is singularly useful, because I can use ChatGPT with Local MCP effectively getting a non-metered AI coding agent.
Seems GPT-5.2 reaches expert level in peer review: 45 scientists took 469 hours evaluating human & AI reviews on 82 papers. “Surprisingly, current AI reviewers are competitive even with the top-rated reviewers in Nature’s official peer review…” though not without weaknesses, so use AI + humans. On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists via Ethan Mollick

The Jamnagar Chokepoint - Data Story

Thu, 01 Jan 2026 03:14:11 +0000

Vivek published an Indian commodity export/import dataset on 31 Dec 2025.

Codex and Claude increased their rate limits for the holiday season, so I had:

Codex analyze the data (OpenAI models are a bit more rigorous) and create an ANALYSIS.md file.
Claude create a visual story based on the analysis. (Claude narrates and visualizes better).

Here is the data story.

Here are the prompts used.

Analyze

I downloaded export-import.parquet from https://github.com/Vonter/india-export-import which has data sourced from the Indian [Foreign Trade Data Dissemination Portal](https://ftddp.dgciskol.gov.in/dgcis/principalcommditysearch.html)

Each row in the dataset represents a trade entry for a single commodity, country, port, year, month, and type (import or export).

- `Commodity` string: Name of the commodity
- `Country` string: Name of the foreign country
- `Port` string: Name of the port in India
- `Year` int32: Year for the import/export activity
- `Month` int32: Month for the import/export activity
- `Type` category: Type of trade (Import or Export)
- `Quantity` int64: Quantity of the commodity
- `Unit` string: Unit for the quantity
- `INR Value` int64: Value of the commodity in INR
- `USD Value` int64: Value of the commodity in USD

Analyze data like an investigative journalist hunting for stories that make smart readers lean forward and say "wait, really?"

- Understand the Data: Identify dimensions & measures, types, granularity, ranges, completeness, distribution, trends. Map extractable features, derived metrics, and what sophisticated analyses might serve the story (statistical, geospatial, network, NLP, time series, cohort analysis, etc.).
- Define What Matters: List audiences and their key questions. What problems matter? What's actually actionable? What would contradict conventional wisdom or reveal hidden patterns?
- Hunt for Signal: Analyze extreme/unexpected distributions, breaks in patterns, surprising correlations. Look for stories that either confirm something suspected but never proven, or overturn something everyone assumes is true. Connect dots that seem unrelated at first glance.
- Segment & Discover: Cluster/classify/segment to find unusual, extreme, high-variance groups. Where are the hidden populations? What patterns emerge when you slice the data differently?
- Find Leverage Points: Hypothesize small changes yielding big effects. Look for underutilization, phase transitions, tipping points. What actions would move the needle?
- Verify & Stress-Test:
  - **Cross-check externally**: Find evidence from the outside world that supports, refines, or contradicts your findings
  - **Test robustness**: Alternative model specs, thresholds, sub-samples, placebo tests
  - **Check for errors/bias**: Examine provenance, definitions, methodology; control for confounders, base rates, uncertainty (The Data Detective lens)
  - **Check for fallacies**: Correlation vs. causation, selection/survivorship Bias (what is missing?), incentives & Goodhart’s Law (is the metric gamed?), Simpson's paradox (segmentation flips trend), Occam’s Razor (simpler is more likely), inversion (try to disprove) regression to mean (extreme values naturally revert), second-order effects (beyond immediate impact), ...
  - **Consider limitations**: Data coverage, biases, ambiguities, and what cannot be concluded
- Prioritize & Package: Select insights that are:
  - **High-impact** (not incremental) - meaningful effect sizes vs. base rates
  - **Actionable** (not impractical) - specific, implementable
  - **Surprising** (not obvious) - challenges assumptions, reveals hidden patterns
  - **Defensible** (statistically sound) - robust under scrutiny

Save your findings in ANALYSIS.md with supporting datasets and code.
This will be taken up by another coding agent to create reports, data stories, visualizations, dashboards, presentations, articles, blog posts, etc.
Ensure that ANALYSIS.md is documented well enough so that all assets are clear, the approach, intent and implications are understandable.

Visualize

I downloaded export-import.parquet from https://github.com/Vonter/india-export-import which has data sourced from the Indian [Foreign Trade Data Dissemination Portal](https://ftddp.dgciskol.gov.in/dgcis/principalcommditysearch.html)

Each row in the dataset represents a trade entry for a single commodity, country, port, year, month, and type (import or export).

- `Commodity` string: Name of the commodity
- `Country` string: Name of the foreign country
- `Port` string: Name of the port in India
- `Year` int32: Year for the import/export activity
- `Month` int32: Month for the import/export activity
- `Type` category: Type of trade (Import or Export)
- `Quantity` int64: Quantity of the commodity
- `Unit` string: Unit for the quantity
- `INR Value` int64: Value of the commodity in INR
- `USD Value` int64: Value of the commodity in USD

Then I had Codex analyze it. The analysis is in ANALYSIS.md.

Find the most intesting insights from ANALYSIS.md and create a data story with supporting visualizations.

Write as a **Narrative-driven Data Story**. Write like Malcolm Gladwell. Think like a detective who must defend findings under scrutiny.

- **Compelling hook**: Start with a human angle, tension, or mystery that draws readers in
- **Story arc**: Build the narrative through discovery, revealing insights progressively
- **Integrated visualizations**: Beautiful, interactive charts/maps that are revelatory and advance the story (not decorative)
- **Concrete examples**: Make abstract patterns tangible through specific cases
- **Evidence woven in**: Data points, statistics, and supporting details flow naturally within the prose
- **"Wait, really?" moments**: Position surprising findings for maximum impact
- **So what?**: Clear implications and actions embedded in the narrative
- **Honest caveats**: Acknowledge limitations without undermining the story

Visualize like The New York Times Interactives. Ensure that all visualizations interactive and provide revelatory insights as well as some kind of delightful experience.

Follow the typography, color & theme, backgrounds, interaction patterns, and animation principles of The Verge's frontends.

Generate a single page index.html + script.js.

Things I Learned - 06 Jul 2025

Sun, 06 Jul 2025 00:00:00 +0000

This week, I learned:

When adding a coding benchmark for LLMs, here’s a question I’d like to add. #benchmark

How do I use Apache Arrow in the browser via cdn.jsdelivr.net to create a .parquet file and download it? Give me minimal working code I can paste in the browser console to test.
LinkedIn has an undocumented link that shows schedules posts at https://www.linkedin.com/share/management/ which redirects to https://www.linkedin.com/feed/?shareActive=true&view=management

Here’s a JS snippet you can paste in the DevTools console of an npm package version page (example) to get a Markdown list showing the versions and dates

copy(
  $$('table[aria-labelledby="version-history"] tbody tr')
    .map((tr) => {
      const a = tr.querySelector("a");
      const date = new Date(tr.querySelector("time").getAttribute("datetime")).toLocaleDateString("en-GB", {
        day: "numeric",
        month: "short",
        year: "numeric",
      });
      return `- [${a.textContent.trim()}](https://npmjs.com${a.getAttribute("href")}): ${date}.`;
    })
    .join("\n"),
);

DuckDB can read JSON APIs! Ref
⭐ When bringing in humans-in-the-loop, applications must make it easier to review and to edit the work.

Embeddings in DuckDB

Sun, 16 Jun 2024 13:06:44 +0000

This article on Using DuckDB for Embeddings and Vector Search by Sören Brunk shows a number of DuckDB features I wasn’t aware of.

DuckDB can read directly from Huggingface datasets
DuckDB can read just the parts of a .parquet file it needs, even over HTTP
DuckDB lets you write custom functions in Python
DuckDB now has a vector similarity search extension

I’ve recently become a DuckDB fan and continue to be impressed.