Here’s roughly what I said in my When Data is for Agents workshop for Fifth Elephant on 7 Jul 2026. Or you can read the detailed AI-generated version if you prefer - it has all the prompts, links, results, etc. I think agents prefer data in a different form than humans. But I don’t know. So, everyone, open ChatGPT (or Claude or whatever), research and ask it! Now, let’s collate them and see the result. Aha! Looks like: ...
I researched the AI policies across 25 universities. In the last 6 months, I conducted sessions at three of these Universities: IIT Madras, Singapore University of Technology and Design, and Ashoka University. Interestingly, these are the three lowest ranked universities in my analysis of AI policies. This is where I’m glad that correlation does not imply causation.
Terence Tao said, “We haven’t done many experiments … large-scale studies where we take a thousand problems and just test them.” So I told Claude: You know my style. Suggest some innovative experiments I could run. The first suggestion was cool! The Polya Audit. Polya’s How to Solve It lists 20 heuristics (work backwards, induction, analogy, etc.). Mathematicians treat these as wisdom. Nobody has ever measured which ones actually work, and on what problem types. ...
The OpenAI text-to-speech cost documentation is confusing. As of 2 Nov 2025: GPT-4o mini TTS costs $0.60 / MTok input and $12.00 / MTok audio output according to the model page and the pricing page. They also estimate this to be ~1.5c per minute - both for input and output. It supports up to 2,000 tokens input. TTS-1 costs $15 / MTok speech generated according to the model page but the pricing page says it's $15 / MChars. No estimate per minute is provided. Is supports up to 4,096 characters input. TTS-1 HD is twice as expensive as TTS-1 I wanted to find the approximate total cost for a typical text input measured per character and token. ...
I asked multiple coding agents and models to build the same app: Create a single-page web app at index.html that beautifully renders a GitHub user profile and activity comprehensively. Pick the ID in the URL ?id=…, default to ?id=torvalds. … and compared their quality, cost, and speed. My observations: Quality variance is the highest. Some models / agents produce great visuals, some average, some fail completely. Cost and time variance are lower among the successful models. About 2X variance in each. ...
I was testing LLMs’ sense of Tamil humor with this quote: Extend this post with more funny Tamil words that end with .ai - mentioning why they’re funny. Chenn.ai is the artificial intelligence capital of India. Kadal.ai Kad.ai Dos.ai Vad.ai Ad.ai Thal.ai Mallig.ai Aratt.ai And finally Podad.ai All spoken in namma bash.ai 😅 The Chinese models didn’t fare well. DeepSeek made up words. Mood.ai - An AI that perfectly captures your mood. Sokk.ai - The AI for when you’re bored. Thanni.ai - A hydration assistant. Qwen too. ...
Evaluate technologies on criteria I care about. Evaluate ... - List all alternatives. - Evaluate them on: - Popular: e.g. # contributors / users - Modern: Updated recently (date of latest release) + regularly (# releases last 12 months) - Admired: e.g. community feedback, expert reviews - Features: breadth of current features - Momentum: how fast are new features being added - Focus: Whet are the recent contributions about? - Cost: Open source self-hostable > Liberal free tier > Low cost > Expensive (mention actual costs) - Docs: Clear docs, examples, tutorials, FAQ, changelog - Light: Low resource usage - Speed: Fast performance, throughput (benchmark if available) - Easy: intuitive, easy installation + usage - CLI tool additional checks: - Scriptable: extensive options for configuration, programmable SDK/API - CLI-configurable: Comprehensive configurability via command line options (without requiring config files) - Bundle: Single-binary > small bundle > large bundle - LLM model additional checks: - Quality: based on evals (name+date+link) - Context: input size, output size - Size: model size, memory usage - Library / framework additional checks: - Adoption: Rapid user growth & adoption in the last 6 months? - Plugins: size of ecosystem - Formats: breadth of input/output formats supported - Platform / services additional checks - Export: How easily, completely, and frequently can we export data in an open format? - Pricing: Is the pricing simple, clear and stable according to users? - Stability: High uptime, low latency, likely to be around for years - Support: responsive support, active community Recommend the top 3 options based on this evaluation. Be rigorous. **Verify and cite sources with dates.** Prefer primary docs, changelogs, and benchmarks. When unsure, state assumptions _up front_. If evidence is weak, **say so** and (if easy) propose ways to gather evidence. List the GitHub repo (if available) for EACH technology evaluated, sorted by stars. Double-check to ensure you don't miss any. Use this format: - [xojs/xo](https://github.com/xojs/xo): Opinionated, zero-config ESLint wrapper with strict defaults. Summary: delightful defaults and smooth “no config” UX; coverage derives from ESLint + curated plugins; development cadence modest; great if you want ESLint’s ecosystem without managing config. - [repo-name](https://github.com/owner/repo-name): 1-line description. Summary of evaluation. GENERAL PREFERENCES (use if applicable) - CLI > Web app > Native app - File formats: Fast + Popular > Open standards > Proprietary - Data-driven configuration/declarative approaches > imperative code - Modular > monolith - Declarative outputs - Browser-first JS + Python - Cross-platform - Stable APIs with migration path
I asked 50 LLMs to multiply 2 numbers: 12 x 12 123 x 456 1,234 x 5,678 12,345 x 6,789 123,456 x 789,012 1,234,567 x 8,901,234 987,654,321 x 123,456,789 LLMs aren’t good tools for math and this is just an informal check. But the results are interesting: Model %Win Q1 Q2 Q3 Q4 Q4 Q6 Q7 openai:o3 86% ✅ ✅ ✅ ✅ ✅ ✅ ❌ openrouter:openai/o1-mini 86% ✅ ✅ ✅ ✅ ✅ ✅ ❌ openrouter:openai/o3-mini-high 86% ✅ ✅ ✅ ✅ ✅ ✅ ❌ openrouter:openai/o4-mini 86% ✅ ✅ ✅ ✅ ✅ ✅ ❌ openrouter:openai/o4-mini-high 86% ✅ ✅ ✅ ✅ ✅ ✅ ❌ deepseek/deepseek-chat-v3-0324 71% ✅ ✅ ✅ ✅ ✅ ❌ ❌ openai/gpt-4.1-mini 71% ✅ ✅ ✅ ✅ ✅ ❌ ❌ openai/gpt-4.5-preview 71% ✅ ✅ ✅ ✅ ✅ ❌ ❌ openai/gpt-4o 71% ✅ ✅ ✅ ✅ ✅ ❌ ❌ openrouter:openai/o3-mini 71% ✅ ✅ ✅ ✅ ✅ ❌ ❌ anthropic/claude-3-opus 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ anthropic/claude-3.5-haiku 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ anthropic/claude-3.7-sonnet:thinking 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ google/gemini-2.0-flash-001 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ google/gemini-2.0-flash-lite-001 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ google/gemini-2.5-flash-preview 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ google/gemini-2.5-flash-preview:thinking 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ google/gemini-2.5-pro-preview-03-25 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ google/gemini-flash-1.5 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ google/gemini-pro-1.5 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ google/gemma-3-12b-it 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ google/gemma-3-27b-it 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ meta-llama/llama-4-maverick 57% ✅ ✅ ✅ ❌ ✅ ❌ ❌ meta-llama/llama-4-scout 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ openai/gpt-4-turbo 57% ✅ ✅ ✅ ✅ ❌ ❌ ❌ openai/gpt-4.1 57% ✅ ✅ ✅ ❌ ✅ ❌ ❌ amazon/nova-lite-v1 43% ✅ ✅ ✅ ❌ ❌ ❌ ❌ amazon/nova-pro-v1 43% ✅ ✅ ✅ ❌ ❌ ❌ ❌ anthropic/claude-3-haiku 43% ✅ ✅ ✅ ❌ ❌ ❌ ❌ anthropic/claude-3.5-sonnet 43% ✅ ✅ ✅ ❌ ❌ ❌ ❌ meta-llama/llama-3.1-405b-instruct 43% ✅ ✅ ❌ ✅ ❌ ❌ ❌ meta-llama/llama-3.1-70b-instruct 43% ✅ ✅ ❌ ✅ ❌ ❌ ❌ meta-llama/llama-3.2-3b-instruct 43% ✅ ✅ ❌ ✅ ❌ ❌ ❌ meta-llama/llama-3.3-70b-instruct 43% ✅ ✅ ❌ ✅ ❌ ❌ ❌ openai/gpt-4.1-nano 43% ✅ ✅ ✅ ❌ ❌ ❌ ❌ openai/gpt-4o-mini 43% ✅ ✅ ✅ ❌ ❌ ❌ ❌ qwen/qwen-2-72b-instruct 43% ✅ ✅ ✅ ❌ ❌ ❌ ❌ anthropic/claude-3-sonnet 29% ✅ ✅ ❌ ❌ ❌ ❌ ❌ deepseek/deepseek-r1 29% ✅ ✅ ❌ ❌ ❌ ❌ ❌ google/gemini-flash-1.5-8b 29% ✅ ✅ ❌ ❌ ❌ ❌ ❌ google/gemma-3-4b-it 29% ✅ ✅ ❌ ❌ ❌ ❌ ❌ meta-llama/llama-3-8b-instruct 29% ✅ ✅ ❌ ❌ ❌ ❌ ❌ meta-llama/llama-3.1-8b-instruct 29% ✅ ❌ ❌ ✅ ❌ ❌ ❌ openai/gpt-3.5-turbo 29% ✅ ✅ ❌ ❌ ❌ ❌ ❌ amazon/nova-micro-v1 14% ✅ ❌ ❌ ❌ ❌ ❌ ❌ meta-llama/llama-2-13b-chat 14% ✅ ❌ ❌ ❌ ❌ ❌ ❌ meta-llama/llama-3-70b-instruct 14% ✅ ❌ ❌ ❌ ❌ ❌ ❌ meta-llama/llama-3.2-1b-instruct 14% ✅ ❌ ❌ ❌ ❌ ❌ ❌ google/gemma-3-1b-it:free 0% ❌ ❌ ❌ ❌ ❌ ❌ ❌ meta-llama/llama-2-70b-chat 0% ❌ ❌ - - ❌ ❌ ❌ Average 96% 86% 66% 58% 24% 10% 0% OpenAI’s reasoning models cracked it, scoring 6/7, stumbling only on the 9-digit multiplication. ...
Simon Willison initiated this very interesting Twitter thread that asks, “What prompt can instantly tell us how good an LLM model is?” The Sally-Anne Test is a popular test that asks: Sally hides a marble in her basket and leaves the room. While she is away, Anne moves the marble from Sally’s basket to her own box. When Sally returns, where will she look for her marble?" ...
Google’s Protocol Buffers is a “language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler” XML is slow and large. There’s no doubting that. JSON’s my default alternative, though it’s a bit large. CSV’s ideal for tabular data, but ragged hierarchies are a bit difficult. I was trying to see if Protocol Buffers would be smaller and faster, at least when using Python. I took JSON as the base, and checked the write speed, read speed and file sizes. Here’s the comparison: ...
I’ve been playing with big data lately. The good part is, it’s easy to get interesting results. The data is so unwieldy that even average value calculations provoke a “Amazing! I didn’t know that,” response (No exaggeration. I heard this from two separate ~ $1bn businesses this month.) The bad part is that calculating even that simple average is slow. For example, take this 40MB file (380MB unzipped) and extract the first column. ...