evaluation on S Anand

Sonnet 4.6 vs MiniMax M2.7

Tue, 24 Mar 2026 17:06:02 +0800

Based on several (i.e. two) recommendations, I subscribed to MiniMax. At $10/month, you get 1,500 requests every 5 hours and 15,000 every week. That’s a LOT!

Using the same prompt I had Claude Code generate two data stories:

The first paragraph, by Claude Sonnet 4.6

The first paragraph, by MiniMax M2.7

Here’s my comparison of the two. It’s partly based on Claude Opus 4.6’s comparison but I felt the same way.

Dimension	Sonnet 4.6	MiniMax M2.7
Narrative quality	Immersive
Content coverage	Comprehensive
Visual design	More varied, ambitious bands, no errors
CSS		Better use of CSS variables
Tooltips	Richer, comprehensive, `data-tip`
Modals/popups	Richer, more types, more details
Animated SVGs	Richer, visually distinctive, sophisticated
Slides	Larger readable grid
Code samples	XML vs JSON-LD side-by-side
External references	Far more authoritative links
Accessibility	ARIA, keyboard, alt text
Generation quality	Clean, no Chinese character artifacts

In other words, Sonnet 4.6 is a clear winner on nearly every dimension.

But the cost factor is too big a difference to ignore. It feels like a 10x difference. So the question probably is: what can I do with a reasonably good model that can generate 10X the quantity at the same price?

(To be fair, GPT 5.4 Mini at 75c/MTok and Gemini 3 Flash at 50c/MTok are not far from MiniMax M2.7 at 30c/MTok - but their code quality seems lower. I generated a Codex - GPT 5.4 Mini version and while it has fewer errors it has even less visual style and narrative quality.)

Computer use feels like a candidate. I used Rodney to research what drives my LinkedIn reach & engagement, and update my SKILL.md.

I could try experimenting with sub-agents, doing bulk analysis (e.g. of code, transcripts, images), data discovery, etc. The crux of these is parallelization - something I have not explored much.

It looks like twe’re entering an era where there are two kinds of use cases: high-quality for the best models, large-scale for the cheap models. The question is: how do I make the most of both?

Source Code

UPDATE: Cheap models (or at least MiniMax M2.7) may be far less useful than I thought. I used MiniMax M2.7 with Claude Code for:

24 Mar 2026: Email analysis. I had it review my 15-year Gramener email archive for key events for a book. But it fetched too few results, so I switched to Codex (GPT 5.4 xhigh).
25 Mar 2026: Capture The Flag. But it couldn’t solve problems, so I switched to Codex (GPT 5.4 xhigh).
25 Mar 2026: Songs download. I had it find popular Tamil songs and download them from YouTube. But the metadata was poor, so I switched to my own song collection.
26 Mar 2026: LEAN proofs. It started making too many basic mistakes (spelling errors in code!) I switched to Copilot (GPT 5.4 xhigh).
29 Mar 2026: Calvin & Hobbes image analysis. It couldn’t even read the images and confidently saw “Hobbes stuck to a baseball bat with Mom & Dad” in a strip that only featured Calvin & Susie.

The main problems are:

It errs confidently. It doesn’t do ROT13 well. It can’t see images. It mis-understands error messages. It assigned my earlier company’s incorporation date (NGIMAGE) as Gramener’s. It made Vijay Sethupathi a lyricist. When a process failed with just 12% coverage, it just continued. It just reported what’s done, not what’s missing.
It’s a slow learner. For picoCTF, it had the pieces but couldn’t assemble them. Claude Code resets the cwd, but it never switched to absolute paths. It mixed uv run with python3. It rewrites, resets or waits instead of diagnosing.

It’s best for simple, single-step tasks. Not where knowledge, accuracy, research matters. When using it, keep tasks small and verify correctness, completeness.

IIM Bangalore PGP Interview Panel

Tue, 17 Mar 2026 06:42:32 +0530

Yesterday, I was part of an IIM Bangalore interview panel at Hyderabad, along with Professor Subhabrata Das and Debajyoti. Panels typically comprise of two faculty and an alumni, and handle 8 interviews in the morning and eight in the evening, though in our case, we had 9 each.

As we arrived, we were given a USB drive with the student’s resume, statement of purpose, and other documents that they had submitted, which included employment contracts, declarations, letters of recommendation, etc., depending on the student. Each interview was approximately 20 minutes. Luckily, Dr Das set a timer for 18, so we didn’t go too far beyond.

There was practically no time to read the documents before each interview, so I used Claude with the following prompt to suggest questions.

In the context of the below, suggest deep, probing questions that will reveal the suitability of the attached candidate for IIM PGP.

Feel free to search online for additional information about the candidate.

…followed by

The instructions email from IIMB
The attachment in the email
Claude’s earlier advice on how to evaluate candidates based on the above. This included suggestions to:
- Probe for internal consistency: “Your SoP says X but your job history suggests Y. Explain.”
- Probe for authenticity: stuff that coaching can’t teach. (It gave candidate-specific questions.)
- Probe for social sensitivity: “You’re from village X. Where does your aspiration help / hurt whihc segments of your village?”
The candidate’s documents (resume, statement of purpose, etc.)

It helped that there were three panelists. That gave me time to screen the questions and understand the good ones - while others asked their questions.

DISADVANTAGE: You have to spend time reading the questions. Not all questions are great. Hallucinations exist (It said one candidate had a claw-back in their employment contract - they didn’t.)
ADVANTAGE: You pick up little-known stuff. One wrote a monthly salary instead of yearly and didn’t know it. The gaps in the resume, the dips in transcripts, etc. surfaced instantly.

Initially, I just read out some questions.
Then I tried using my own judgement.
Then I went back to Claude’s questions.
It took a while to learn how to use it well.

The interview questions idea came from Claude. It also suggested:

Transcribe real time and pop-up questions. I wish I could and I’m sure it’ll happen soon.
Review scores & transcript with Claude. I did almost exactly that!

I told GitHub Copilot CLI (I had credits) running Claude 4.6 Sonnet (a sensible model):

You are evaluating candidates for the PGP Program at IIM Bangalore.
Read interview-guidelines.md fully to understand the process and criteria.

Then, go through each candidate’s folder. These contain:

notes.md: interview notes

documents.md: documents submitted (converted from the PDF in that folder).

Primarily based on the notes (using documents.md for reference - no need to read PDFs), evaluate all candidates on the criteria mentioned in interview-guidelines.md.

Plan like an expert interview evaluator first. In this context, first think about:

What patterns would an expert in this field check / recognize that beginners would miss?

What questions would an expert ask that a beginner would not know to?

What problems / failures would an expert anticipate that beginners may not be aware of?

How would an expert analyze this? At each step, explain what they are looking for and why.

Document and plan your process in eval-copilot-claude/plan.md.
Then, analyze students (use sub-agents as required) and document your analysis and evaluation in eval-copilot-claude/evaluation.md.

It reviewed all 18 candidates summarizing their background, work-experience, and interview scores along with strengths, concerns, and notable moments, with an evaluation summary.

This serves as a good cross-check, I think. So I told it to:

Compare the ratings against my ratings (and notes) in README.md.
Sort the candidates based on the difference between my ratings vs your (normalized / scaled from 1-10) ratings.
For the biggest differences, analyze the reasons for the differences.
Specifically, what might I have missed in the candidate that I should take a closer look at?
Append this to eval-copilot-claude/evaluation.md.

I ran this on Copilot as well as Claude. We agreed on the best candidate. No question about that. Interesting!

This revealed some interesting observations. I seem to:

Ignore social insensitivity and artistic ability. Most likely because I lack both. Note to self: I’m blind to, and hence devalue, what I’m not good at.
Value genuineness far more than analytics. I rated one candidate with terrible analysis skills as only a marginal reject. Claude rejected them outright. This is true for another candidate, too. But I rejected fakers outright.
I calibrate for preparation. I expect more from those with more experience, with more interview preparation, etc. Not a bad thing - just an observation.

In short:

AI is great for preparation. I should make it easier to feed it context.
AI is great at post-mortems. I need to budget time for this.
AI can help during discussions. I need to figure out the technology for this.

Cracking online exams with coding agents

Fri, 13 Mar 2026 15:37:19 +0800

An effective way to solve online exams is to point a coding agent at it.

I use that on my Tools in Data Science course in two ways:

As a test case of my code. If my agent can solve it, good: I set the question correctly.
As a test of student ability. If it can’t, good: it’s a tough question (provided I didn’t make a mistake).

For PyConf, Hyderabad, my colleague built a Crack the Prompt challenge. Crack it and you get… I don’t know… goodies? A job interview? Leaderboard bragging rights?

I told Codex:

Use the browser to visit https://crack-the-prompt.straivedemo.com/ and solve it using the email ID root.node@gmail.com and GitHub handle sanand0

After 4 minutes, it told me:

The answers to all three prompt-engineering questions
The code has a bug - so no one can submit anyway
The prompts are hidden on the server-side (making it a bit harded to hack)
But you can skip levels via the API - level-locking is front-end only
… and a whole bunch of interesting things.

When I asked Claude to write about the process in Matt Levine’s style, it included an interesting lesson.

The Victorians had the same problem. They designed elaborate entrance exams for the civil service because they wanted to identify people with the capacity for careful, systematic thinking. Then someone invented the civil service exam prep industry, and suddenly the exam was measuring preparation rather than capacity.

The challenge was about the process – about developing the instincts, the questioning strategies, the ability to read AI behavior like a poker tell. That’s the thing you can’t automate. Or rather, it’s the thing you can automate, which means it’s no longer a skill worth developing, which means we need to think about what skill we’re actually trying to cultivate when we design these challenges.

EXACTLY. What’s the skill we’re trying to cultivate? When will it be outdated?

From now on, when testing, I’m going to write down “What skill is this really testing?” That’s good enough a start.

The Future of Work with AI

Wed, 11 Mar 2026 13:56:58 +0800

I often research how the world will change with AI by asking AI. Today’s session was informative. I asked Claude, roughly

Economics changes human behavior. As intelligence cost falls to zero, here are some changes in my behavior [I listed these].

Others will have experienced behavioral changes too. Search online and synthesize behavioral changes.

It said this.

🟡 People spend time on problem framing & evaluation. AI can execute the middle. (I’m OK at this. Need to do more framing + evaluation.)
🟢 People don’t plan, they just build. (I’m prototyping a lot.)
🟢 People build personal data & context. (I’m mining my digital exhaust.)
🔴 People queue work for agents, delegating into the future. (I’m not. I need to do far more of this.)
🟢 People shift from searching to asking for answers. (I do this a lot, e.g. this post.)
🟡 People are AI-delegating junior jobs and developing senior level taste early. (Need to do more.)
🟡 People treat unresolved emotions as prompts. (Need to do more.)

Rough legend: 🟢 = Stuff I know. 🟡 = I kind-of know. 🔴 = New learning.

My next question was:

Using economics, psychology, sociology, etc. predict new behavior changes and implications.

It said: the bottleneck shift to trust, taste, and attention.

🟢 Taste is a skill. Few people will deepen craft engagement (reading, cooking, music, …) but most won’t, creating a gap.
- Fuzzier areas (e.g. art, philosophy, unnamed ones) have more value than verifiable ones (e.g. code, science) since they’re harded to automate.
🟢 Context is an asset. People & companies will record decisions, reasoning, relationship histories, .. to create (marketplace-tradeable) assets.
🟡 Trust is an asset. People who stand behind AI output will create value in law, medicine, education, etc. Consulting will restructure around this. This will require investments # in:
- Observability
- Human augmentation - gracefully degrading without harm when humans inevitably falter
- Synthetic practice
- Cryptographic provenance
- liability regimes that internalize tail risk
🟢 AI-less is exercise. Few people will deliberately train without AI, and will be valuable in niches (sports, emergencies, status-symbols). Exams / filters will partially screen for this.
🟡 AI-less feels broken. AI-speed will be expected in services and interactions. Relationships are assets that become valuable and premium.
🔴 Orgs re-form. Org sizes fall. But size has advantages (trust, brand, connects) so new orgs will form with teams evaluated on outcome (portfolio, ROI) and trust (network, ratings).
🔴 Experience becomes luxury. Non-reproducible experiences become expensive. Provable authenticity commands a premium.
🔴 Two-speed world. Some places (countries, companies, colleges, communities) become more AI friendly. Capital, talent and productivity will concentrate here.

What implication will these have on the nature of work?

Historically, apprenticeship (execution) preceded mastery (judgement). Now, execution is free. That messes things.

🟡 How do we develop judgement? Simulators, red-teaming, AI-free tests, …?
🔴 What will managers do? Less coordination & oversight. More judgement (validate output not process), motivation, accountability. Like a film director, not supervisor.
🟡 How will we hire/pay? More outcome-based hire/pay. More freelancing, portfolio or reputation based hiring. Long-term retainers reserved for trust.
🔴 How will we describe our work? Less about tasks (which AI does) and more about who you are, where you fit, what you contribute.

So work will organize around:

Trust: context, judgement, accountability
Presence: caring, building with hands, performing, connecting
Direction: framing, evaluating, curating

… and less around translation (execution, coordination, oversight).

Tools in Data Science - Jan 2026

Thu, 29 Jan 2026 07:33:22 +0800

My Tools in Data Science course is available publicly, with a few changes from last year.

First, I removed all the content! Last year, Claude generated teaching material using my prompts. But what’s the point? I might as well give students the prompts directly. They can tweak it to their needs.

This time, TDS shares the questions needed to learn a topic. Any AI will give you good answers.

Second, it focuses on what AI does NOT do well. Coding syntax? Who cares. Basic analysis? ChatGPT can do that. In fact, each question now has an “Ask AI” button that dumps the question into your favorite AI tool. Just paste the answer and move on.

But, these questions (hopefully) teach you where AI fails. Putting tools together, prompting well, debugging and evaluating the output, etc. These matter more.

Third, it’s easier to audit. Anyone can take the course, even outside IITM. You can join the public Google Group for announcements. All questions are discussed publicly on GitHub.

So, check the new version out, learn, and please share feedback!

Google AI Tools List

Sun, 25 Jan 2026 10:35:32 +0530

Google has released a huge number of AI tools. Not all are useful, but some are quite powerful. Here’s a list of the tools ChatGPT could find.

🟢 = I find it good. 🟡 = Not too impressive. 🔴 = Avoid.

Assistants, research, and knowledge work
- 🟢 Gemini is Google’s main AI assistant app. Use it as a meeting-prep copilot: paste the agenda + last email thread, ask for “3 likely objections + crisp rebuttals + 5 questions that sound like I did my homework.”
  - 🟢 Gemini Deep Research is Gemini’s agentic research mode that browses many sources (optionally your Gmail/Drive/Chat) and produces multi-page reports. Use it to build a client brief with citations (market, competitors, risks), then reuse it for outreach or a deck outline.
  - 🟢 Gemini Canvas turns ideas (and Deep Research reports) into shareable artifacts like web pages, quizzes, and simple apps. Use it to convert a research report into an interactive explainer page your team can share internally.
  - 🟢 Gemini Agent is an experimental “do multi-step tasks for me” feature that can use connected apps (Gmail/Calendar/Drive/Keep/Tasks, plus Maps/YouTube). Use it to plan a week of customer check-ins: “find stalled deals, draft follow-ups, propose times, and create calendar holds-show me before sending.”
- 🟢 NotebookLM is a source-grounded research notebook: it answers from your uploaded sources and can generate Audio Overviews. Use it to turn a messy folder of PDFs into a decision memo + an “AI podcast” you can listen to while walking.
- 🟡 Pinpoint (Journalist Studio) helps explore huge collections of docs/audio/images with entity extraction and search. Use it for internal investigations / audit trails: upload contracts + emails, then trace every mention of a vendor and its linked people/locations.
- 🟢 Google AI Mode exposes experimental Search experiences (including AI Mode where available). Use it for rapid competitive scans: run the same query set weekly and track what changed in the AI-generated summaries vs links.
- Project Mariner is a Google Labs “agentic” prototype aimed at taking actions on your behalf in a supervised way. Use it to prototype a real workflow (e.g., “collect pricing from 20 vendor pages into a table”) before you invest in automating it properly.
Workspace and “AI inside Google apps”
- 🟢 Google Workspace with Gemini brings Gemini into Gmail/Docs/Sheets/Drive, etc. Use it to turn a weekly leadership email into: (1) action items per owner, (2) a draft reply, and (3) a one-slide summary for your staff meeting.
- Google Vids is Workspace’s AI-assisted video creation tool. Use it to convert a project update doc into a 2-3 minute narrated update video for stakeholders who don’t read long emails.
- Gemini for Education packages Gemini for teaching/learning contexts. Use it to generate differentiated practice: same concept, three difficulty levels + a rubric + common misconceptions.
Build: developer + agent platforms
- 🟢 Google AI Studio is the fast path to prototyping with Gemini models and tools. Use it to build a “contract red-flagger”: upload a contract, extract clauses into structured JSON, and generate a risk report you can paste into your workflow.
- Firebase Studio is a browser-based “full-stack AI workspace” with agents, unifying Project IDX into Firebase. Use it to ship a real internal tool (auth + UI + backend) without local setup, then deploy with Firebase/Cloud Run.
- 🟢 Jules is an autonomous coding agent that connects to your GitHub repo and works through larger tasks on its own. E.g. give it “upgrade dependencies, fix the failing tests, and open a PR with a clear changelog,” then review it like a teammate’s PR instead of doing the grind yourself.
  - Jules Tools (CLI) is a command-line interface for running and monitoring Jules from your terminal or CI. E.g. pipe a TODO list into “one task per session,” auto-run nightly maintenance (lint/format/test fixes), and have it open PRs you can batch-review in the morning
  - Jules API lets you programmatically trigger Jules from other systems. E.g. when a build fails, your pipeline can call the API with logs + stack trace, have Jules propose a fix + tests, and post a PR link back into Slack/Linear for human approval
- Project IDX > Firebase Studio is the transition site if you used IDX. Use it to keep your existing workspaces but move to the newer Studio flows (agents + Gemini assistance).
- Genkit is an open-source framework for building AI-powered apps (workflows, tool use, structured output) across providers. Use it to productionize an agentic workflow (RAG + tools + eval) with a local debugging UI before deployment.
- Stax is Google’s evaluation platform for LLM apps (prompts, models, and end-to-end behaviors), built to replace “vibe testing” with repeatable scoring. E.g. codify your product’s rubric (tone, factuality, refusal correctness, latency), run it against every prompt/model change, and block releases when key metrics regress
- SynthID is DeepMind’s watermarking approach for identifying AI-generated/altered content. E.g. in an org that publishes lots of content, watermark what your tools generate and use detection as part of provenance checks before external release
  - SynthID Text is the developer-facing tooling/docs for watermarking and detecting LLM-generated text. E.g. watermark outbound “AI-assisted” customer emails and automatically route them for review if they’re about regulated topics
- Responsible Generative AI Toolkit is Google’s “safeguards” hub: watermarking, safety classifiers, and guidance to reduce abuse and failure modes. E.g. wrap your app with layered defenses (input filtering + output moderation + policy tests) so one jailbreak prompt doesn’t become a security incident
- Vertex AI Agent Builder is Google Cloud’s platform to build, deploy, and govern enterprise agents grounded in enterprise data. Use it to build a customer-support agent that can read policy docs, query BigQuery, and write safe responses with guardrails.
- Gemini Code Assist is Gemini in your IDE (and beyond) with chat, completions, and agentic help. Use it for large refactors: ask it to migrate a module, generate tests, and propose PR-ready diffs with explanations.
- PAIR Tools is Google’s hub of practical tools for understanding/debugging ML behavior (especially interpretability and fairness). E.g. before launch, run “slice analysis + counterfactual edits + feature sensitivity” to find where the model breaks on real user subgroups
- LIT (Learning Interpretability Tool) is an interactive UI for probing models on text/image/tabular data. E.g. debug prompt brittleness by comparing outputs across controlled perturbations (tense, style, sensitive attributes) and visualizing salience/attribution to see what the model is actually using
- What-If Tool is a minimal-coding tool to probe model predictions and fairness. E.g. manually edit a single example into multiple “what-if” counterfactuals and see which feature flips the decision, then turn that into a targeted data collection plan
- Facets helps you explore and visualize datasets to catch skew, outliers, and leakage early. E.g. audit a training set for missingness and subgroup imbalance, then fix data before you waste time “tuning your way out” of a data problem
- 🟡 Gemini CLI brings Gemini into the terminal with file ops, shell commands, and search grounding. Use it as a repo-native “ops copilot”: “scan logs, find the regression, propose the patch, run tests, and summarize.”
- 🟡 Antigravity (DeepMind) is positioned as an agentic development environment. Use it when you want multiple agents running tasks in parallel (debugging, refactoring, writing tests) while you supervise.
- Gemini for Google Cloud is Gemini embedded across many Google Cloud products. Use it for cloud incident triage: summarize logs, hypothesize root cause, and generate the Terraform/IaC fix.
Create: media, design, marketing, and “labs” tools
- Google Labs is the hub for many experiments (Mixboard, Opal, CC, Learn Your Way, Doppl, etc.). Use it as your “what’s new” page-many tools show up here before they become mainstream.
- 🟡 Opal builds, edits, and shares AI mini-apps from natural language (with a workflow editor). Use it to create a repeatable analyst tool (e.g., “take a company name > pull recent news > summarize risks > draft outreach”).
- 🟡 Mixboard is an AI concepting canvas/board for exploring and refining ideas. Use it to run a structured ideation sprint: generate 20 variants, cluster them, then turn the top 3 into crisp one-pagers.
- Pomelli is a Labs marketing/brand tool that can infer brand identity and generate on-brand campaign assets. Use it to produce a month of consistent social posts from your website + a few product photos.
- 🟡 Stitch turns prompts/sketches into UI designs and code. Use it to go from a rough wireframe to React/Tailwind starter code you can hand to an engineer the same day.
- 🟡 Flow is a Labs tool aimed at AI video/story production workflows (built around Google’s gen-media stack). Use it to create a pitch sizzle reel quickly: consistent characters + scenes + a simple timeline.
- Whisk is a Labs image tool focused on controllable remixing (subject/scene/style style workflows). Use it for fast, art-directable moodboards when text prompting is too loose.
- ImageFX is Google Labs’ image-generation playground. Use it to iterate brand-safe visual directions quickly (e.g., generate 30 “hero image” variants, pick 3, then refine).
- VideoFX is the Labs surface for generative video (Veo-powered). Use it to prototype short looping video backgrounds for product pages or events.
- MusicFX is the Labs music generation tool. Use it to generate royalty-free stems (intro/outro/ambient) for podcasts or product videos.
- Doppl is a Labs try-on style experiment/app. Use it to sanity-check creative wardrobe ideas before you buy, or to mock up “virtual merch” looks for a campaign.
- 🟢 Gemini Storybook creates illustrated stories. Use it to generate custom reading material for a specific learner’s interests (and adjust reading level/style).
- TextFX is a Labs-style writing creativity tool (wordplay, transformations, constraints). Use it to generate 10 distinct “hooks” for the same idea before you write the real piece.
- GenType is a Labs experiment for AI-generated alphabets/type. Use it to create a distinctive event identity (custom letterforms) without hiring a type designer for a one-off.
Science, security, and “serious AI”
- AlphaFold Server provides AlphaFold structure prediction as a web service. Use it to test protein/ligand interaction hypotheses before spending lab time or compute on deeper simulations.
- Google Threat Intelligence uses Gemini to help analyze threats and triage signals. Use it to turn a noisy alert stream into a prioritized, explainable threat narrative your SOC can act on.
Models
- 🟡 Gemma is DeepMind’s family of lightweight open models built from the same tech lineage as Gemini. E.g. run a small, controlled model inside your VPC for narrow tasks (classification, extraction, safety filtering) when sending data to hosted LLMs is undesirable
- 🟡 Model Garden is Vertex AI’s catalog to discover, test, customize, and deploy models from Google and partners. E.g. shortlist 3 candidate models, run the same eval set, then deploy the winner behind one standardized platform with enterprise controls
- Vertex AI Studio is the Google Cloud console surface for prototyping and testing genAI (prompts, model customization) in a governed environment. E.g. keep “prompt versions + test sets + pass/fail criteria” together so experiments become auditable artifacts, not scattered chats
- Model Explorer helps you visually inspect model graphs so you can debug conversion/quantization and performance issues. E.g. compare two quantization strategies and pinpoint exactly which ops caused a latency spike or accuracy drop before you deploy
- Google AI Edge is the umbrella for building on-device AI (mobile/web) with ready-to-use APIs across vision, audio, text, and genAI. E.g. ship an offline, privacy-preserving feature (document classification or on-device summarization) so latency and data exposure don’t depend on the network
  - Google AI Edge Portal benchmarks LiteRT models across many real devices so you don’t guess performance from one phone. E.g. test the same model on a spread of target devices and pick the smallest model/config that consistently hits your FPS/latency target
- TensorFlow Playground is an interactive sandbox for understanding neural networks. E.g. use it to teach or debug intuitions—show how regularization, feature interactions, or class imbalance changes decision boundaries in minutes
- Teachable Machine lets anyone train simple image/sound/pose models in the browser and export them. E.g. prototype an accessibility feature (custom gesture or sound trigger) fast, then export the model to a small web demo your stakeholders can try
Directories (“where to discover the rest”)
- Google DeepMind Products & Models (Gemini, Veo, Astra, Genie, etc.)-best “canonical list” of what exists.
- Google Labs Experiments directory-browse by category (develop/create/learn) to catch smaller experiments you didn’t know to search for.
- Experiments with Google is a gallery of interactive demos (many AI) that’s great for prompt/data literacy and workshop “aha” moments. E.g. curate 5 experiments as a hands-on “AI intuition lab” for your team so they learn failure modes by playing, not by reading docs

Tue, 01 Jul 2025 06:46:10 +0000

LLMs are smarter than us in many areas. How do we control them?

It’s not a new problem.

VC partners evaluate deep-tech startups.
Science editors review Nobel laureates.
Managers manage specialist teams.
Judges evaluate expert testimony.
Coaches train Olympic athletes.

… and they manage and evaluate “smarter” outputs in many ways:

Verify. Check against an “answer sheet”.
Checklist. Evaluate against pre-defined criteria.
Sampling. Randomly review a subset.
Gating. Accept low-risk work. Evaluate critical ones.
Benchmark. Compare against others.
Red-team. Probe to expose hidden flaws.
Double-blind review. Mask identity to curb bias.
Reproduce. Re-running gives the same output?
Consensus. Ask many. Wisdom of crowds.
Outcome. Did it work in the real world?

For example, you can apply them to:

Vibe coding: Non-programmers might glance at lint checks (Checklist) and see if it works (Outcome).
LLM image designs: Developers might check if a few images look good (Sampling) and check a few marketers (Consensus).
LLM news articles: An journalist might run a Checklist, a Double-blind review with experts, and Verify critical facts (Gating).

You already know many of these. You learnt them in Auditing. Statistics. Law. System controls. Policy analysis. Quality engineering. Clinical epidemiology. Investigative journalism. Design critique.

Worth brushing up on these skills. They’re more important in the AI era.

ChatGPT: https://chatgpt.com/share/6863733f-4ebc-800c-ad3f-a2b472d9e9ca

How To Control Smarter Intelligences

Tue, 01 Jul 2025 06:37:40 +0000

LLMs are smarter than us in many areas. How do we manage them?

This is not a new problem.

VC partners evaluate deep-tech startups.
Science editors review Nobel laureates.
Managers manage specialist teams.
Judges evaluate expert testimony.
Coaches train Olympic athletes.

… and they manage and evaluate “smarter” outputs in many ways:

Verify. Check against an “answer sheet”.
Checklist. Evaluate against pre-defined criteria.
Sampling. Randomly review a subset.
Gating. Accept low-risk work. Evaluate critical ones.
Benchmark. Compare against others.
Red-team. Probe to expose hidden flaws.
Double-blind review. Mask identity to curb bias.
Reproduce. Re-running gives the same output?
Consensus. Aggregate multiple responses. Wisdom of crowds.
Outcome. Did it work in the real world?

For example:

Vibe coding: Non-programmers might glance at lint checks (Checklist) and see if it works (Outcome).
LLM image designs: Developers might check if a few images look good (Sampling) and check a few marketers (Consensus).
LLM news articles: An journalist might run a Checklist, a Double-blind review with experts, and Verify critical facts (Gating).

You already know many of these. You learnt them in Auditing. Statistics. Law. System controls. Policy analysis. Quality engineering. Clinical epidemiology. Investigative journalism. Design critique.

Worth brushing up skills. They’re more important in the AI era.

It's so easy to fool yourself

Sun, 30 Mar 2025 01:45:05 +0000

I asked 4 LLMs to “Suggest catchy rhyming slogans to promote journaling.” Here are the suggestions:

GPT 4.5
- Write it right, day or night!
- Reflect, inspect, journaling perfect!
- Express to impress, journaling’s best!
- Clear your mind, write and unwind!
- Pen your zen, again and again!
- Daily pages, inner sages!
- Take a pause, write your cause!
- Emotions flow, journals grow!
- Write, ignite, your inner light!
- Ink it, think it, don’t just blink it!
- Pen your quest, ease your stress!
- From confusion to conclusion—journal your solution!
- Keep a book, take a look, watch your growth hook!
- Thoughts on paper, worries taper!
- Write to heal, make it real!
Claude 3.7 Sonnet
- Write it down, turn that frown upside down!
- Pen your thoughts, connect the dots.
- Journal every day, keep the stress away.
- Ink your mind, treasures you’ll find.
- Write and reflect, your future self will thank you direct.
- Scribble and doodle, your mind will stay in the noodle.
- Journal with flair, show yourself you care.
- Jot it in ink, give yourself time to think.
- Fill the page, release the rage.
- Write your story, capture the glory.
- Daily writing, life enlightening.
- Express in a line, your day will shine.
- Thoughts on paper, life feels safer.
- Write with heart, that’s the smart start.
- Journaling’s neat, makes life complete.
Gemini 2.5 Pro
- Thoughts unclear? Write them down here.
- Let the ink help you think.
- Untangle your mind, one line at a time.
- To make things clear, keep your journal near.
- When your head’s in a knot, give journaling a shot.
- Feeling stressed? Put your pen to the test.
- Write it down, let worries drown.
- Ease your mind, leave troubles behind.
- Let your feelings take flight, just sit down and write.
- Turn the page on inner rage.
- Write to know, watch yourself grow.
- Explore your soul, take control.
- Pen and page, turn a new age.
- Discover your way, write every day.
- Know your heart? Make journaling your art.
- Capture the day in your own way.
- Make moments last, write down the past.
- Don’t let memories stray, write them today.
- Reflect and connect, gain self-respect.
- Life moves fast? Make memories last.
- Get in the flow, write and grow.
- Pen in hand, understand.
- Daily lines, clear designs.
- Just take note, stay afloat.
- Write your heart, make a fresh start.
Grok 3:
- Write your day, let thoughts play!
- Pen in hand, your mind expands!
- Journal your way, seize the day!
- Feel the groove, let words move!
- Scribe your soul, make it whole!
- Ink flows free, just be me!
- Page by page, unlock your sage!
- Write it down, wear the crown!

At first glance, GPT 4.5 didn’t impress me. Claude 3.7 Sonnet did. I also didn’t like Gemini 2.5 Pro, but Grok was great.

Grok 3 > Claude 3.7 Sonnet > Gemini 2.5 Pro > GPT 4.5.

But it’s hard to compare a dozen quotes at once. So I made a small quotes arena app to help me pick my favorites. It shows me random pairs of quotes and asks which I like more.

To my surprise, after answering 30+ “games” in the arena, I found that based on my preferences:

Claude 3.7 Sonnet > Gemini 2.5 Pro > GPT 4.5 > Grok 3.

That was weird. I thought I liked Grok’s results a lot. I continued till I answered 50+ games. Then I found that:

Grok 3 > GPT 4.5 > Gemini 2.5 Pro > Claude 3.7 Sonnet.

That’s the exact opposite of the previous result.

Honestly, I’m depressed. I’ve learnt 3 things:

I can’t judge stuff at a glance.
But I think I can (especially with code.)
Even when evaluating carefully, my preferences are unstable.

Nothing has shaken my confidence more in recent times. I cannot trust my judgement. I need written evals. Badly.

LLMs still do not locate bounding boxes well

Fri, 01 Nov 2024 16:20:03 +0000

I sent an image to over a dozen LLMs that support vision, asking them:

Detect objects in this 1280x720 px image and return their color and bounding boxes in pixels. Respond as a JSON object: {[label]: [color, x1, y1, x2, y2], …}

None of the models did a good-enough job. It looks like we have some time to go before LLMs become good at bounding boxes.

I've given them a subjective rating on a 1-5 scale below.

Model	Positions	Sizes
gemini-1.5-flash-001	🟢🟢🟢🔴🔴	🟢🟢🟢🟢🔴
gemini-1.5-flash-8b	🟢🟢🟢🔴🔴	🟢🟢🟢🔴🔴
gemini-1.5-flash-002	🟢🟢🔴🔴🔴	🟢🟢🟢🔴🔴
gemini-1.5-pro-002	🟢🟢🟢🔴🔴	🟢🟢🟢🟢🔴
gpt-4o-mini	🟢🔴🔴🔴🔴	🟢🟢🔴🔴🔴
gpt-4o	🟢🟢🟢🟢🔴	🟢🟢🟢🟢🔴
chatgpt-4o-latest	🟢🟢🟢🟢🔴	🟢🟢🟢🟢🔴
claude-3-haiku-20240307	🟢🔴🔴🔴🔴	🟢🟢🔴🔴🔴
claude-3=5-sonnet-20241022	🟢🟢🟢🔴🔴	🟢🟢🟢🔴🔴
llama-3.2-11b-vision-preview	🔴🔴🔴🔴🔴	🔴🔴🔴🔴🔴
llama-3.2-90b-vision-preview	🟢🟢🟢🔴🔴	🟢🟢🟢🔴🔴
qwen-2-vl-72b-instruct	🟢🟢🟢🔴🔴	🟢🟢🔴🔴🔴
pixtral-12b	🟢🟢🔴🔴🔴	🟢🟢🟢🔴🔴

I used an app I built for this.

Here is the original image along with the individual results.

Update

Adding gridlines with labeled axes helps the LLMs. (Thanks @Bijan Mishra.) Here are a few examples:

How does Gemini process videos?

Thu, 24 Oct 2024 08:47:21 +0000

The Gemini documentation is clear:

The File API service extracts image frames from videos at 1 frame per second (FPS) and audio at 1Kbps, single channel, adding timestamps every second. These rates are subject to change in the future for improvements in inference.

Note: The details of fast action sequences may be lost at the 1 FPS frame sampling rate. Consider slowing down high-speed clips for improved inference quality.

Individual frames are 258 tokens, and audio is 32 tokens per second. With metadata, each second of video becomes ~300 tokens, which means a 1M context window can fit slightly less than an hour of video.

To ask questions about time-stamped locations, use the format MM:SS, where the first two digits represent minutes and the last two digits represent seconds.

But on this ThursdAI episode: Oct 17 - Robots, Rockets, and Multi Modal Mania…, at 1:00:50, Hrishi says

I don’t think it’s a series of images anymore because when I talk to the model and try to get some concept of what it’s perceiving, it’s no longer a series of images.

If that’s the case, it’s a huge change. So I tested it with this video.

This video has 20 numbers refreshing at 4 frames per second.

When I upload it to AI Studio, it takes 1,316 tokens. This is close enough to 258 tokens per image (no audio). So I’m partly convinced that Gemini still processing videos at 1 frame per second.

Then, I asked it to Extract all numbers in the video using Gemini 1.5 Flash 002 as well as Gemini 1.5 Flash 8b. In both cases, the results were: 2018, 85, 47, 37, 38.

These are frames 2, 6, 10, 14, 18 (out of 20). So, clearly Gemini is still sampling at about 1 frame per second, starting somewhere between 0.25 or 0.5 seconds.