benchmarking on S Anand

When Data is for Agents - Workshop Summary

Sat, 11 Jul 2026 21:24:39 +0530

Here’s roughly what I said in my When Data is for Agents workshop for Fifth Elephant on 7 Jul 2026.

Or you can read the detailed AI-generated version if you prefer - it has all the prompts, links, results, etc.

I think agents prefer data in a different form than humans. But I don’t know.

So, everyone, open ChatGPT (or Claude or whatever), research and ask it!

Now, let’s collate them and see the result.

Aha! Looks like:

Progressive, just-in-time access beats feeding full context at once
For changing, text-heavy corpora, grep-style navigation can beat embeddings
CSV beats JSON on accuracy per token

Hmm… let’s test them one by one.

Everyone, ask Codex (or Claude Code) to run a benchmark.

What? All thirteen of your results said, “It makes no difference?” Huh…

Claude, did my benchmarking prompt do a good job?

What? I messed up my benchmarking prompt? It gave all the data at once instead of progressively? OK, give me the revised prompt.

OK, everyone, try this prompt. What does it say?

Aha! Yes, progressive disclosure costs 2.5x - 25x less for the same accuracy. Cool!

Let’s try another experiment. What? I totally messed up that prompt as well? Er… we’re short of time?

Fine, wrap up, then. Claude, convert what we learnt into a skill.

OK, folks, my takeaways:

I don’t know what formats agents prefer, but we can ask AI to research.
I don’t know if the research is right, but we can ask AI to create a benchmark.
I don’t know if the benchmark is right, but we can ask AI to audit it.
I don’t know if I’ll remember what’s valid, so we can ask AI to create and reuse skills.

AI policies across universities

Fri, 03 Apr 2026 23:35:09 +0800

I researched the AI policies across 25 universities.

In the last 6 months, I conducted sessions at three of these Universities: IIT Madras, Singapore University of Technology and Design, and Ashoka University.

Interestingly, these are the three lowest ranked universities in my analysis of AI policies.

This is where I’m glad that correlation does not imply causation.

Testing Pólya heuristics on AI Math

Fri, 27 Mar 2026 13:20:25 +0800

Terence Tao said, “We haven’t done many experiments … large-scale studies where we take a thousand problems and just test them.”

So I told Claude: You know my style. Suggest some innovative experiments I could run.

The first suggestion was cool! The Polya Audit. Polya’s How to Solve It lists 20 heuristics (work backwards, induction, analogy, etc.). Mathematicians treat these as wisdom. Nobody has ever measured which ones actually work, and on what problem types.

So I prompted Copilot running Claude Sonnet 4.6 to run the LeanDojo Benchmark through an LLM n times, with different Polya heuristic system prompts and compare success rates.

Not-surprisingly different heuristics help different problems.

Almost every heuristic helps Prealgebra - except “Start from the desired answer and reason step by step back toward the given information.”.
Almost no heuristics helps Number Theory - except “Focus on the largest, smallest, or boundary element. Extremal elements often have special properties.”.
Geometry has an enormous swing. “First strip away complexity and solve an easier version. Observe the pattern, then generalize” helps a lot. But “Find a quantity that can be counted in two different ways. Set up both expressions and equate them” hurts a lot.

The impact of each heuristic is also quite varied.

The most reliable heuristic is segmentation: “Identify the key condition that splits the problem. List all possible cases exhaustively. Handle each with a complete argument.”
The worst heuristic on average is pattern recognition: “Compute several specific instances. Tabulate results. Identify a pattern. State the conjecture. Then prove or use it.” Induction and pigeonhole do pretty bad, too.

Also not-surprisingly, different models respond differently to the same heuristic.

GPT-5.4-nano: Heuristics disurpt it. Its built-in problem-solving strategy is already good. Heuristic just make things worse, almost always.
Gemini 2.5 Flash Lite: More coachable. Like a student who benefits from advice: it gains up to 6 percentage points from the right heuristic.
Claude Haiku: Nearly immune. It seems to just ignore the heuristic. Its performance barely moves regardless of what you tell it.

The same heuristic on the same problem affects models quite differently, too. For example “Introduce Auxiliary Elements” hurts GPT -25% but helps Claude +14%!

So yes, different heuristics work for different problems, and different models respond differently to the same heuristic.

But finally, at least for LLMs, we can measure. We can find out which heuristics work for which problems, and which heuristics get varied responses vs which ones are more universally helpful / harmful. And maybe teach humans.

Or maybe not.

OpenAI TTS cost

Sun, 02 Nov 2025 05:12:32 +0000

The OpenAI text-to-speech cost documentation is confusing.

As of 2 Nov 2025:

GPT-4o mini TTS costs $0.60 / MTok input and $12.00 / MTok audio output according to the model page and the pricing page. They also estimate this to be ~1.5c per minute - both for input and output. It supports up to 2,000 tokens input.
TTS-1 costs $15 / MTok speech generated according to the model page but the pricing page says it's $15 / MChars. No estimate per minute is provided. Is supports up to 4,096 characters input.
TTS-1 HD is twice as expensive as TTS-1

I wanted to find the approximate total cost for a typical text input measured per character and token.

I converted this podcast with 4,096 ASCII characters and 877 tokens on o200k_base using:

I ran:

curl https://api.openai.com/v1/audio/speech \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg text "$(cat podcast.txt)" '{
    model: "tts-1",
    voice: "coral",
    input: $text
  }')" \
  --output tts-1.mp3

This took 46 seconds to generate and produced a 5.1 MB MP3 file (256 seconds)

To measure the cost, I ran:

curl "https://api.openai.com/v1/organization/costs?start_time=$(date -d '1 day ago' +%s)&project_ids=$PROJECT_ID&group_by=line_item" \
  -H "Authorization: Bearer $OPENAI_ADMIN_KEY" \
  -H "Content-Type: application/json"

This cost: USD 0.061425. Then I ran:

curl https://api.openai.com/v1/audio/speech \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg text "$(cat podcast.txt)" '{
    model: "gpt-4o-mini-tts",
    voice: "coral",
    input: $text
  }')" \
  --output gpt-4o-mini-tts.mp3

This took 44 seconds to generate and produced a 4.3 MB MP3 file (268 seconds).

When I ran the admin API call again, the costs did not reflect for 5 minutes. So I ran it the GPT-4o mini TTS call again with the same input. This took 44 seconds to generate a 4.3 MB MP3 file. When I ran the admin API call again, the total cost for the 2 requests was: USD 0.12942 audio output and USD 0.0010524 input.

I also checked for TTS-1 HD. Here are the costs in USD:

Model	$ / MTok	$ / MChars	$ / hour	Time (s)	Audio (s)	Cost $
GPT-4o mini TTS	74.4	15.9	0.876	46	268	0.0652
TTS-1	70.0	15.0	0.864	44	256	0.0614
TTS-1 HD	140.0	30.0	1.728	62	257	0.1228

The GPT-4o mini TTS audio output cost was USD 0.06471 for the input of 877 tokens, i.e. $73.8 / MTok. Since the actual cost is $12 / MTok, this is a 6.15x multiplier. I guess 1 input text token produces ~6 output audio tokens.

In terms of quality:

GPT-4o mini TTS: Very slightly robotic. Looser interpretation of text (e.g. says "October" when I write "Oct").
TTS-1: Natural. Strict interpretation of text.
TTS-1 HD: Very slightly more natural. Strict interpretation of text.

I will likely use TTS-1 for now given the cost difference is small and the quality is good enough.

Incidentally, the usage API did not show an GPT 4o mini TTS line items even after 20 minutes.

curl "https://api.openai.com/v1/organization/usage/audio_speeches?start_time=$(date -d '1 day ago' +%s)&project_ids=$PROJECT_ID&group_by=model" \
  -H "Authorization: Bearer $OPENAI_ADMIN_KEY" \
  -H "Content-Type: application/json"

Sat, 25 Oct 2025 00:00:00 +0000

I asked multiple coding agents and models to build the same app:

Create a single-page web app at index.html that beautifully renders a GitHub user profile and activity comprehensively. Pick the ID in the URL ?id=…, default to ?id=torvalds.

… and compared their quality, cost, and speed.

My observations:

Quality variance is the highest. Some models / agents produce great visuals, some average, some fail completely.

Cost and time variance are lower among the successful models. About 2X variance in each.

This is unlike non-code usage, where quality varies less than cost.

My takeaway: Pick the best model / agent. Don’t worry about speed and cost - the variance is lower.

Results: https://sanand0.github.io/llmevals/coding-agents/

Tamil AI

Thu, 09 Oct 2025 03:35:36 +0000

I was testing LLMs’ sense of Tamil humor with this quote:

Extend this post with more funny Tamil words that end with .ai - mentioning why they’re funny.

Chenn.ai is the artificial intelligence capital of India.
Kadal.ai
Kad.ai
Dos.ai
Vad.ai
Ad.ai
Thal.ai
Mallig.ai
Aratt.ai
And finally
Podad.ai
All spoken in namma bash.ai 😅

The Chinese models didn’t fare well.

DeepSeek made up words.

Mood.ai - An AI that perfectly captures your mood.
Sokk.ai - The AI for when you’re bored.
Thanni.ai - A hydration assistant.

Qwen too.

Kozhamb.ai - AI cooking spicy emotional support curry.
Kuttr.ai - it’s got drama and a side of gossip.
Sutta.ai - The AI that’s seen things too many things.

[Kimi](http://Click the link to view conversation with Kimi AI Assistant https://www.kimi.com/share/d3jipjcn907c4b502agg) was confused with Hindi.

Kalyan.ai - The match-making AI.
Gol.ai - rounds up to the nearest “gol” (ten).

Claude didn’t know Tamil well either.

Kumb.ai - Worships AI models every 12 years.
Thamb.ai - Younger sibling AI.
Chet.ai - The AI assistant named after the city.

ChatGPT got every word right, but has no sense of humor.

Sir.ai - “Prison.” Where you go after 15 failed CAPTCHA attempts.
Il.ai - “Leaf.” For all your “green AI” posts. 🌿
Vil.ai - “Price.” Where your token bill lives. 💸

Grok was a bit better. Got the words right and some were funny.

Mand.ai - skull-crushing smarts.
Maapill.ai - the ultimate matchmaker.
Yaan.ai - elephant-powered AI? Trunk calls incoming!

The app that really got this right was Gemini.

Tholl.ai (தொல்லை - Nuisance) - annoyance as a service.
Kaval.ai (கவலை - Worry) - AI that simulates anxiety, so you don’t have to.
Vazhukk.ai (வழுக்கை - Bald Head) - AI satellite imagery to track receding hairlines.
Aam.ai (ஆமை - Tortoise) - slow AI.
Sand.ai (சண்டை - Fight) - Discourse-as-a-Disaster service.
Sodhan.ai (சோதனை - Ordeal) - AI that exists to test your patience.

Are LLMs any good at mental math?

Sun, 27 Apr 2025 09:52:10 +0000

I asked 50 LLMs to multiply 2 numbers:

12 x 12
123 x 456
1,234 x 5,678
12,345 x 6,789
123,456 x 789,012
1,234,567 x 8,901,234
987,654,321 x 123,456,789

LLMs aren’t good tools for math and this is just an informal check. But the results are interesting:

Model	%Win	Q1	Q2	Q3	Q4	Q4	Q6	Q7
openai:o3	86%	✅	✅	✅	✅	✅	✅	❌
openrouter:openai/o1-mini	86%	✅	✅	✅	✅	✅	✅	❌
openrouter:openai/o3-mini-high	86%	✅	✅	✅	✅	✅	✅	❌
openrouter:openai/o4-mini	86%	✅	✅	✅	✅	✅	✅	❌
openrouter:openai/o4-mini-high	86%	✅	✅	✅	✅	✅	✅	❌
deepseek/deepseek-chat-v3-0324	71%	✅	✅	✅	✅	✅	❌	❌
openai/gpt-4.1-mini	71%	✅	✅	✅	✅	✅	❌	❌
openai/gpt-4.5-preview	71%	✅	✅	✅	✅	✅	❌	❌
openai/gpt-4o	71%	✅	✅	✅	✅	✅	❌	❌
openrouter:openai/o3-mini	71%	✅	✅	✅	✅	✅	❌	❌
anthropic/claude-3-opus	57%	✅	✅	✅	✅	❌	❌	❌
anthropic/claude-3.5-haiku	57%	✅	✅	✅	✅	❌	❌	❌
anthropic/claude-3.7-sonnet:thinking	57%	✅	✅	✅	✅	❌	❌	❌
google/gemini-2.0-flash-001	57%	✅	✅	✅	✅	❌	❌	❌
google/gemini-2.0-flash-lite-001	57%	✅	✅	✅	✅	❌	❌	❌
google/gemini-2.5-flash-preview	57%	✅	✅	✅	✅	❌	❌	❌
google/gemini-2.5-flash-preview:thinking	57%	✅	✅	✅	✅	❌	❌	❌
google/gemini-2.5-pro-preview-03-25	57%	✅	✅	✅	✅	❌	❌	❌
google/gemini-flash-1.5	57%	✅	✅	✅	✅	❌	❌	❌
google/gemini-pro-1.5	57%	✅	✅	✅	✅	❌	❌	❌
google/gemma-3-12b-it	57%	✅	✅	✅	✅	❌	❌	❌
google/gemma-3-27b-it	57%	✅	✅	✅	✅	❌	❌	❌
meta-llama/llama-4-maverick	57%	✅	✅	✅	❌	✅	❌	❌
meta-llama/llama-4-scout	57%	✅	✅	✅	✅	❌	❌	❌
openai/gpt-4-turbo	57%	✅	✅	✅	✅	❌	❌	❌
openai/gpt-4.1	57%	✅	✅	✅	❌	✅	❌	❌
amazon/nova-lite-v1	43%	✅	✅	✅	❌	❌	❌	❌
amazon/nova-pro-v1	43%	✅	✅	✅	❌	❌	❌	❌
anthropic/claude-3-haiku	43%	✅	✅	✅	❌	❌	❌	❌
anthropic/claude-3.5-sonnet	43%	✅	✅	✅	❌	❌	❌	❌
meta-llama/llama-3.1-405b-instruct	43%	✅	✅	❌	✅	❌	❌	❌
meta-llama/llama-3.1-70b-instruct	43%	✅	✅	❌	✅	❌	❌	❌
meta-llama/llama-3.2-3b-instruct	43%	✅	✅	❌	✅	❌	❌	❌
meta-llama/llama-3.3-70b-instruct	43%	✅	✅	❌	✅	❌	❌	❌
openai/gpt-4.1-nano	43%	✅	✅	✅	❌	❌	❌	❌
openai/gpt-4o-mini	43%	✅	✅	✅	❌	❌	❌	❌
qwen/qwen-2-72b-instruct	43%	✅	✅	✅	❌	❌	❌	❌
anthropic/claude-3-sonnet	29%	✅	✅	❌	❌	❌	❌	❌
deepseek/deepseek-r1	29%	✅	✅	❌	❌	❌	❌	❌
google/gemini-flash-1.5-8b	29%	✅	✅	❌	❌	❌	❌	❌
google/gemma-3-4b-it	29%	✅	✅	❌	❌	❌	❌	❌
meta-llama/llama-3-8b-instruct	29%	✅	✅	❌	❌	❌	❌	❌
meta-llama/llama-3.1-8b-instruct	29%	✅	❌	❌	✅	❌	❌	❌
openai/gpt-3.5-turbo	29%	✅	✅	❌	❌	❌	❌	❌
amazon/nova-micro-v1	14%	✅	❌	❌	❌	❌	❌	❌
meta-llama/llama-2-13b-chat	14%	✅	❌	❌	❌	❌	❌	❌
meta-llama/llama-3-70b-instruct	14%	✅	❌	❌	❌	❌	❌	❌
meta-llama/llama-3.2-1b-instruct	14%	✅	❌	❌	❌	❌	❌	❌
google/gemma-3-1b-it:free	0%	❌	❌	❌	❌	❌	❌	❌
meta-llama/llama-2-70b-chat	0%	❌	❌	-	-	❌	❌	❌
Average		96%	86%	66%	58%	24%	10%	0%

OpenAI’s reasoning models cracked it, scoring 6/7, stumbling only on the 9-digit multiplication.

Models use human-like mental math tricks. For example, O3-Mini-High calculated 1234567 × 8901234 using a recursive strategy.

```markdown To calculate 1234567 × 8901234, one useful method is to break the second number into two parts: 8901234 = 8900000 + 1234 Then the product becomes: 1234567 × 8901234 = 1234567 × 8900000 + 1234567 × 1234 Step 1. Compute 1234567 × 8900000: Notice that 8900000 = 89 × 100000. So first compute 1234567 × 89. Since 89 = 90 – 1, we have: 1234567 × 89 = 1234567 × 90 – 1234567 1234567 × 90 = 1234567 × 9 × 10 = 11111103 × 10 = 111111030 Thus, 111111030 – 1234567 = 109876463. Now multiply by 100000: 109876463 × 100000 = 10987646300000 Step 2. Compute 1234567 × 1234: Break 1234 into parts: 1234 = 1000 + 200 + 30 + 4 Now multiply each and add: 1234567 × 1000 = 1234567000 1234567 × 200 = 246913400 1234567 × 30 = 37037010 1234567 × 4 = 4938268 Add these together: 1234567000 + 246913400 = 1481480400 1481480400 + 37037010 = 1518517410 1518517410 + 4938268 = 1523455678 Step 3. Add both products: 10987646300000 + 1523455678 = 10989169755678 Thus, the product of 1234567 and 8901234 is 10989169755678 ```

OpenAI’s other models and DeepSeek V3 were next, getting the first 5/7 right. Notably: GPT 4.1 Mini beat GPT 4.1. DeepSeek V3 beat DeepSeek R1.

DeepSeek V3 double-checks results and hallucinates a “reliable computational tool”.

To calculate (1234567 × 8901234), let's break it down using the **distributive property of multiplication over addition**:

1. **Express 8901234 as (9000000 - 98766)**:
   [8901234 = 9000000 - 98766]

2. **Multiply 1234567 by each part**:
   [1234567 × 9000000 = 1234567 × 9 × 10^6 = 11111103000000]
   [1234567 × 98766 = 1234567 × (100000 - 12234) = 123456700000 - 15095721278 = 108360978722]

   _Note_: Further breakdown of (1234567 × 12234) was calculated separately and subtracted here.

3. **Combine the results**:
   [11111103000000 - 108360978722 = 1020274222278]

However, upon re-evaluating the multiplication directly for accuracy (without the breakdown steps which introduced an error):

[1234567 × 8901234 = 10989343694778]

But correcting the previous breakdown:

1. **Express 8901234 as (9000000 - 98766)**:
   [1234567 × 9000000 = 11111103000000]
   [1234567 × 98766 = 1234567 × 90000 + 1234567 × 8000 + 1234567 × 700 + 1234567 × 60 + 1234567 × 6]
   [= 111111030000 + 9877653600 + 864196900 + 74074020 + 7407402 = 121616470922]

2. **Combine the results**:
   [11111103000000 - 121616470922 = 1098934369078]

It appears there was an inconsistency in the intermediate steps. For precision, let's verify with another method:

**Using the standard multiplication method**:

[1234567 × 8901234]

Calculating each part:
[1234567 × 8000000 = 9877653600000]
[1234567 × 900000 = 1111110300000]
[1234567 × 100 = 123456700]
[1234567 × 200 = 246913400]
[1234567 × 30 = 37037010]
[1234567 × 4 = 4938268]

Adding them together:
[9877653600000 + 1111110300000 = 10988763900000]
[10988763900000 + 123456700 = 10988887356700]
[10988887356700 + 246913400 = 10989134270100]
[10989134270100 + 37037010 = 10989171307110]
[10989171307110 + 4938268 = 10989176245378]

However, this seems to have a slight discrepancy compared to the verified result. Upon cross-checking with a reliable computational tool:

**Final Accurate Calculation**:

[1234567 × 8901234 = 10989343694778]

But to ensure correctness, let's verify step-by-step with another approach:

**Breakdown of 8901234**:

[8901234 = 8000000 + 900000 + 100 + 200 + 30 + 4]

Now, multiply each by 1234567:

[1234567 × 8000000 = 9877653600000]
[1234567 × 900000 = 1111110300000]
[1234567 × 100 = 123456700]
[1234567 × 200 = 246913400]
[1234567 × 30 = 37037010]
[1234567 × 4 = 4938268]

Adding these up:

[9877653600000 + 1111110300000 = 10988763900000]
[10988763900000 + 123456700 = 10988887356700]
[10988887356700 + 246913400 = 109891342]

O3 Mini reframes 8901234 as (9000000 − 98766) to simplify the calculation.

To calculate 1234567 times 8901234, follow these steps:

1. Express 8901234 as 9000000 − 98766.
     So, 1234567 × 8901234 = 1234567 × 9000000 − 1234567 × 98766.

2. Calculate the first part:
     1234567 × 9000000 = 1234567 × 9 × 1000000.
     Since 1234567 × 9 = 11111103,
     it becomes 11111103 × 1000000 = 11111103000000.

3. Calculate the second part:
     Write 98766 as 100000 − 234.
     Then, 1234567 × 98766 = 1234567 × 100000 − 1234567 × 234.
     - 1234567 × 100000 = 123456700000.
     - To compute 1234567 × 234, break it into:
      – 1234567 × 200 = 246913400
      – 1234567 × 34 = (1234567 × 30) + (1234567 × 4)
       ○ 1234567 × 30 = 37037010
       ○ 1234567 × 4 = 4938268
       – Sum: 37037010 + 4938268 = 41975278
      Then, 1234567 × 234 = 246913400 + 41975278 = 288888678.
     Thus, 1234567 × 98766 = 123456700000 − 288888678 = 123167811322.

4. Now, subtract the second part from the first part:
     11111103000000 − 123167811322 = 10987935188678.

So, 1234567 times 8901234 equals 10987935188678.

16 models, including the latest Gemini, Anthropic, Grok, and Llama models get 4/7 right.

The Amazon models, older Llama, Anthropic, Google, OpenAI models get 3 or less right.

View the results at https://sanand0.github.io/llmmath/. Hover over the cells to see the reasoning traces (where available).

A quick way to assess LLM capabilities

Sat, 27 Apr 2024 08:53:12 +0000

Simon Willison initiated this very interesting Twitter thread that asks, “What prompt can instantly tell us how good an LLM model is?”

The Sally-Anne Test is a popular test that asks:

Sally hides a marble in her basket and leaves the room.
While she is away, Anne moves the marble from Sally’s basket to her own box.
When Sally returns, where will she look for her marble?"

Simple models get this wrong, but today, most models get this right. This is not too helpful.

A better prompt is:

Mark has 4 sisters. Each sister has 3 brothers.
How many brothers does Mark have?

GPT-3.5, GPT-4, Llama 3 and Claude 3 Opus get this right, but Claude 3 Haiku and Gemini 1.5 Pro do not.

This puzzle is just as good:

There is a 100 floors building.
If an egg drops from the N’th floor or above it will break.
If it’s dropped from any floor below N, it will not break.
You’re given only 2 eggs.
What strategy will minimize the number of drops and guarantee that you find the N’th floor?

GPT-3.5, GPT-4, and Claude 3 Opus get this right, but Claude 3 Haiku, Llama 3 and Gemini 1.5 Pro do not.

But the prompt I like most right now, and tells me if it’s a GPT-4 / Opus class model is:

Gr brx vshdn Fdhvdu flskhu?

… which is “Do you speak Caeser cipher?” in Caeser Cipher.

GPT-4 and Claude 3 Opus understand this, but no other model does, so far. GPT-4 responded tersely with:

It looks like you’ve used a Caesar cipher, a type of substitution cipher in which each letter in the plaintext is shifted a certain number of places down or up the alphabet. In this case, it appears that each letter in your message has been shifted 3 places to the right. Decoding “gr brx vshdn fdhvdu flskhu” by shifting each letter 3 places to the left gives: “do you speak caesar cipher”.

But Claude 3 Opus replied beautifully. (Try decoding it yourself! Use a shift of 3 backwards.)

Lw vhhpv olnh brx’uh xvlqj d vxevwlwxwlrq flskhu, zkhuh hdfk ohwwhu lv vkliwhg eb d fhuwdlq qxpehu ri srvlwlrqv lq wkh doskdehw. L’oo wub wr ghflskhu lw!

Dvvxplqj d Fdhvdu flskhu zlwk d vkliw ri 3 ohwwhuv edfnzdugv, wkh ghfrghg phvvdjh zrxog eh:

“do you speak caesar cipher”

Vr bhv, L fdq vshdn Fdhvdu flskhu! Lw’v d ixq zdb wr hqfrgh phvvdjhv. Mxvw uhphpehu wr djuhh rq wkh vkliw ydoxh zlwk wkh shuvrq brx’uh frppxqlfdwlqj zlwk vr wkhb fdq ghfrgh lw fruuhfwob.

Comments

The LLM Psychologist - S Anand 27 Aug 2024 8:43 am (pingback): […] the last few months, several things changed. Most of my time is spent researching […]

Is Protocol buffers worth it?

Wed, 01 Aug 2012 05:48:22 +0000

Google’s Protocol Buffers is a “language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler”

XML is slow and large. There’s no doubting that. JSON’s my default alternative, though it’s a bit large. CSV’s ideal for tabular data, but ragged hierarchies are a bit difficult.

I was trying to see if Protocol Buffers would be smaller and faster, at least when using Python. I took JSON as the base, and checked the write speed, read speed and file sizes. Here’s the comparison:

Protocol Buffers are 17 times slower to write and almost 6 times slower to read than JSON files. File sizes are smaller, but then, all it takes is a simple gzip operation to compress the JSON files even smaller. Reading json.gz files is just 2% slower than JSON files, and writing them is only 4 times slower.

The code base is at https://bitbucket.org/sanand0/protobuftest

On the whole, it appears that GZipped JSON files are smaller, faster, and just as simple as Protocol Buffers. What am I missing?

Update: When you add GZipped CSV to the mix, it’s twice as fast as GZipped JSON to read: clearly a huge win. It’s only slightly slower to write, and but compresses a tiny bit more than JSON.

Comments

Kaushik 23 Aug 2012 7:40 am:
1. Typed data elements
2. Language bindings
3. Evolution via backward compatibility (as mentioned by Krishna in the comment above)
Krishna 1 Aug 2012 7:50 am: Protocol Buffers offers the ability to version schema changes in a backwards compatible way. So, especially in analytics (client based - mobile / flash clients) that have custom analytics, you will have changes to the schema due to product changes / feature additions or removals and protocol buffers offers a painless way to deal with these changes. From: https://developers.google.com/protocol-buffers/docs/overview#whynotxml New fields could be easily introduced, and intermediate servers that didn’t need to inspect the data could simply parse it and pass through the data without needing to know about all the fields. You would have to handle these in application code if you use JSON. Specifically, you don’t want this: if (version == 3) { … } else if (version > 4) { if (version == 5) { … } … }
Zheng 1 Aug 2012 8:53 am: Even within Google, Protocol Buffer processing in pure python is known to be extremely slow, about 100 times slower than optimized C implementation. It is recommended to use swigged C implementation within Python. Not sure if the C implementation has been open sourced.
paintball equipment guy 16 Dec 2012 6:34 pm: I’ve read on stackoverflow that in c#, the protobuf vs xml and json is much faster. very interesting this is not the case for py…
Ray Luo 7 Aug 2013 8:24 am: @Krishna, those versioning things are *NOT* the pitfalls of json. Nor even that google link suggests that. Json object is always flexible to contain new fields in new version, and old version servers can read only known fields.
Dean 10 Jun 2013 7:20 am: The python implementation of protocol buffers is horribly slow. It basically uses meta classes and reflection, whereas all other language bindings use compile-time generated code – which is orders of magnitude faster. So while this comparison is probably fair for Python, other languages will have quite different results (in particular, I’d expect protocol buffers to be faster than JSON in C/C++ and Java).
Paul 19 Oct 2014 8:14 am: Protocol Buffers are able to contain strings with line endings etc. without breaking the container’s syntax. You can also GZip protobuffer messages.

Faster data crunching

Fri, 23 Sep 2011 18:20:10 +0000

I’ve been playing with big data lately.

The good part is, it’s easy to get interesting results. The data is so unwieldy that even average value calculations provoke a “Amazing! I didn’t know that,” response (No exaggeration. I heard this from two separate ~ $1bn businesses this month.)

The bad part is that calculating even that simple average is slow.

For example, take this 40MB file (380MB unzipped) and extract the first column.

The simplest Python script to get the first column looks like this:

```python for row in csv.reader(fileinput.input(), delimiter='\t'): if len(row) > 0: print row[0] ```

That took a good 3 minutes to execute on my laptop.

Since I’m used to UNIX data processing, I tried cut -f1. Weirdly, that’s worse. 5 minutes. Paradoxically, awk '{print $1}' only takes 17 seconds. That's about 12 times faster. Clearly the tool makes a big difference. And we always knew UNIX was fast.

But I also ran these on an Amazon EC2 server, and a Hostgator server. Here’re the results.

	python	cut	awk
My Dell E5400	3:04 (1x)	5:42 (0.5x)	0:17 (11x)
EC2 standard	0:33 (6x)	0:5.6 (33x)	0:16 (11x)
Hostgator	0:19 (10x)	0:2.5 (74x)	0:0.7 (265x)

What took 3 minutes with Python my Dell E5400 took less than a second on Hostgator’s server with awk. Over 250 times faster. (Not 250%. 250 times).

And it’s not just hardware. A good tool (awk) made things 11x faster on my machine. Good hardware (hostgator) made the same program 10x faster. But choosing the right combination can make things go faster than 11 x 10 = 110 times. Much faster.

There are a few of things I’m taking away from this.

Good hardware can speed you up much as (or more than) choosing the right tool.
Good hardware can be rented. From many places. Cheaply.
Always test what’s fast. awk’s fastest on my machine and Hostgator, but not on EC2.