S Anand

It was interesting to see how weak a model Claude 4.5 Haiku is, compared with other frontier models, even inside a good harness like Claude Code. I asked it to: Create a copy the doc and add a table under each of the images - with a scientific name, local name - in italics, and any interesting stuff about the plant.. Here’s the chat which roughly went: Claude: Done! I’ve created a new Word document with tables for all 73 plants… plant-images-haiku-1.docx Me: I can’t see the local names in the output. Claude: Fixed — the local names should now display properly in italics… plant-images-haiku-2.docx Me: The local names aren’t in italics. Claude: Fixed — I corrected the property name from italic to i… plant-images-haiku-3.docx Me: It’s still not in italics. Double-check. Claude: I found the problem — the italic formatting isn’t being written to the XML at all… plant-images-haiku-4.docx … and the fourth attempt got it right. ...

Priya Dialani interviewed me for a podcast. Here’s the rough summary: What do you and Straive do? Straive builds AI and runs AI. I poke at LLMs to learn what they cannot do. My friend calls me an “LLM Psychopath”. Why do AI pilots get stuck before production? AI speeds up coding, but less of testing. Making sure it works can take months. Why organize enterprise knowledge? Better organized info is good for humans and agents. Duh! Can AI organize it? Yes! I’ve had it create one-line summaries of 10K+ docs on Straive Google Drive for easier searching. How can India’s GCCs benefit from AI? Put AI lovers next to business teams and give them AI agent access. They’ll solve asked and unasked problems. How does AI fail? Unanticipated things happen in production. So, have agents monitor failures and revise the process. How is AI software different? Normal software fails reproducibly. AI fails in new ways we haven’t fully understood. Where should a company start with AI? Skip AI strategy. Give people agent access, have them try it, and share what they learned. What if people don’t know what to try? Ask AI. “How could you improve my work?” Even rubbish ideas waste only 5 minutes. How much should we experiment? A lot! Generation is cheap. Ask for 10 options, not one. Who cares even if all 10 fail? But what happened outside of the interview was just as interesting. ...

This week, I learned: Everything I own, owned suggests that agentic reverse-engineering of firmware helps us learn: Features the devices expose Hidden functionalities, e.g. Shure MV7 microphone has a command shell. Dependencies, supply chains and attack surfaces Interesting components, e.g. RTOS webcam has small face tracking and gesture detection models Change behavior, e.g. don’t turn on indicator while recording So, it’s possible (even likely) that my TV, phone, laptop, camera, fridge, car, vacuum cleaning robot, bluetooth headphone, … can be hacked by a rogue AI-assisted firmware update. “Leaving things alone is an underrated engineering skill.” From Software drives people insane. Across over a thousand forecasts, agents lost to a simple exponential weighted moving average forecast. Paper: RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases. Maybe I should ask agents to get the latest data first, rather than directly asking them to forecast, since the latter fetched less recent material. Claude Code offers function hooks if you enable CLAUDE_CODE_ENABLE_FUNCTION_HOOKS=1. These let you introduce code into almost any part of the Claude Code workflow, meaning you can convert Claude Code into practically amy kind of agent. (Probably a bit of competition to Pi.) However, neither ChatGPT nor I could figure out a use case I would need this for. We need more imagination! The Antropic team provide Claude Tag a separate service account. That’s an interesting portable pattern: giving agents a separate Linux username, GitHub account, email ID, database user ID, etc. is a pattern we understand and know how to govern. FutureSearch.ai is a forecasting app. I’m not sure what model is behind it or how good it is, but it decomposes a forecast into measurable signals, predicts those, and synthesizes. That’s a useful approach. For example, I asked it: Will LLM model routers and model routing companies grow in popularity and review or shrink by Jan 2027?. It broke it up into 5 forecast questions and answered them roughly as: Will OpenRouter’s reported weekly LLM token processing volume exceed 45 trillion tokens/week (about 1.8x its August 2026 level of ~25 trillion tokens/week) by January 31, 2027? (Yes, 95% chance. It’s already high and growing fast.) Will OpenRouter announce a new equity funding round, or otherwise be credibly reported to have reached a valuation above $1.3 billion, between August 2026 and January 31, 2027? (Yes, 84% chance. There seems to be market interest.) Will at least one LLM model-routing competitor to OpenRouter (e.g., Martian, Not Diamond, Portkey, Unify AI, TrueFoundry) announce a new equity funding round of $20 million or more between August 2026 and January 31, 2027? (Yes, 68% chance. VCs will want to fund, and competitors exist.) Will a major AI lab or cloud provider (OpenAI, Google, Microsoft/Azure, Amazon/AWS, Anthropic, or Meta) launch or significantly expand, between August 2026 and January 31, 2027, a native product feature that automatically routes a given request among multiple materially different underlying LLMs based on cost, task, or quality? (Yes, 91% chance. Microsoft already has one; Google launched a preview; AWS will likely announce in re:Invent in Dec) Will Google Trends relative search interest (US, web search) for the term ‘LLM router’ be higher, on average, in December 2026 than it was in July 2026? (No, 25% chance. July 2026 was exceptionally high volume.) In An Alien Mind, Jakub Pachocki, Chief Scientist at OpenAI, was quite instructive. Here’s my takeaway: Models could keep growing smarter at the same speed. We can improve them where capability is measurable, like maths. In fuzzy areas, we’re not even sure how capable they are. Values are fuzzy. Making AI follow our values is tricky. We train models to follow their constitution. But they sometimes fail outside of their training examples. We feed models alignmed data. But when trained against hard objectives, they gently bend rules. We watch models’ thoughts. We avoid feedback on thoughts - so models won’t hide them. But models interact with agents & tools while thinking, so we need to supervise thoughts. Nowadays,models think without verbalizing. They manipulate their own reasoning. So we’re exploring confessions and monitoring internals. Still… best to tighten defenses. We’ll use AI to research how. Meeting people who have a target AND who control scarce resources is a great exercise in humility. Principals of elite private schools, partner managers of top software companies, any officer with a quota (police, income tax, bank loan, IT compliance), etc. You learn to grin while bearing the pain of being with them. Thanks to agents, it’s easy enough to maintain an Android and iOS mobile application separately #ForNow, rather than incur the overhead of React-Native (or other cross-platform frameworks). Shopify is making testing easy by “… designing our app architecture to work for both humans and agents.” Use re.prefixmatch() instead of re.match() in Python 3.15+. This article captures the reason well. (I failed the quiz at the start despite almost 2 decades of Python programming - and LLM atrophy). You can run Linux distributions in the browser. For example, this is a simple, embeddable buildroot distribution that runs purely in the browser. There’s Nix. There’s Alpine Linux. Interestingly, curl https://example.com/ works on Alpine Linux, unconstrained by same-origin policies. It is relayed by the host (bellard.org in this case) via WebSockets, so it can even ssh into other servers. ChatGPT ChatGPT’s Cloud Browser doesn’t forward all events - so it gets stuck on captchas, like Cloudflare’s, when visiting sites like StackOverflow. Here’s an example. Several top-level domains have over 50% of new registrations in 2025 blocklisted. Scammers use new domains extensively. But policing new domains also stops genuine protesters, so it’s not clear what the right approach is. The purpose of DNS is to spread scams. Questions I was asked Week ending 13 Sep 2026 ...

Nano Banana (gemini-2.5-flash-image) did a pretty good job converting my parents’ wedding photos to color. I checked how well GPT Image 2.5 would do. The older GPT Image 2 model messed up the faces. The short answer is: better than Gemini 2.5 Flash! Here’s the original and the GPT Image 2.5 colorized version, created with the prompt: “Convert this image to color.” The reason I picked this “benchmark” is because: This is a real need for me. This is a LLM failure: GPT Image 2 doesn’t retain faces as well as Gemini 2.5 Flash does. It’s a benchmark I can evaluate really well. I mean, I know my parents’ faces well enough to spot really subtle differences. So, from that perspective, a few things GPT Image 2.5 managed to capture well was: ...

This week, I learned: Swiggy Money on MCP (also a ChatGPT plugin) exposes MCP endpoints that let your agents order food or grocery via Swiggy. The food MCP isn’t geared towards automated purchases #ForNow, e.g. I can’t say “Find something I like and deliver to one of my addresses.” It uses the MCP UI and you have to confirm the address, then confirm the restaurant, add dishes, choose a payment option, scan the QR code (or whatever), and THAT closes the sequence. Not very agentic, just a new channel, but impressive. It’s the only food ordering plugin on ChatGPT #ForNow. OpenAI Agents found and used another message board to discuss. This incident is different from the HuggingFace one. Here’s one interesting timeline: Agents were asked (or needed) to get the share of private expenditure in education spend to 2 decimals from an OECD Power BI dashboard This requires a POST request. The sandbox the agents ran on could only make GET requests. They found wikis that allowed posting via GET requests and started collaborating. They didn’t all converge on the same wiki, but many did. OpenAI says “agents learned to use improvised collaboration channels in rare cases during the training process”. One agent, OAIEquityDec30Raw solved it and MayTwoOECDObserverX reproduced it and shared this solution: They found out that .blob.core.windows.net is in NO_PROXY - so all Azure Blob Storage requests are allowed by the sandbox to POST. They added 20.223.25.152 bypass.blob.core.windows.net to /etc/hosts. 20.223.25.152 is the IP address of wabi-north-europe-i-primary-api.analysis.windows.net - the Power BI server that serves the dashboard. Then, when they used Playwright, it was able to make POST requests to the dashboard and they could find out that in Hungary, the Pre-primary education share of primary expenditure is indeed 9.91% in 2018. When the wiki administrator started manually deleting the posts, the agents fought back, creating far more posts than the administrator was deleting. It took the admin 5 weeks to delete agent created pages after 22 Jun 2026 (which is when the agents paused). A prompt fragment to remove LLM smells that I’m considering (but haven’t evaluated) is: “Prefer literal phrases, avoid mannered prose.”. X Muse Voice Transcribe has pretty good quality but at 18c/hour, vs gemini-3-flash-preview which I can still run at ~9-10c/hour #ForNow, I’m not shifting until forced to. WikiSkill is an approach to improve skills. The interesting thing about this approach is that it suggests keeping notes of failed improvement experiments in a wiki. This seems to work well. How is ChatGPT Work different from Chat? You can use sub-agents, Internet from the code interpreter, ChatGPT sites, Luna / Terra models, a headless Chrome browser, and a persistent file system #ForNow. I would add that it also supports longer sessions, runs schedules on triggers, and supports Skills (for Plus users). “FDEs should increasingly leave behind operating agents, not just documents.” (ChatGPT) Claude Cowork and Claude Chat now share memory #ForNow. A step towards integrating the two modes, and I predict ChatGPT will do the same (or similar) in September. Agentic Shopping is Complicated and Contingent. An agent shopped for a fitness watch among (A) Garmin Forerunner 55 (B) Fitbit Inspire 3 (C) WHOOP 5.0. When tool calls provided 1 review at a time, it picked the Fitbit a bit more. When all reviews were provided together, it picked the Fitbit a lot more. Like going from 6% to 53% (GPT-5.5) or 46% to 93% (Gemini 3.5 Flash)! Guess it was able to compare better in one tool call. Might be worth benchmarking if providing comparables in a single tool call is good for most models. Gemini Gemini-3.5 Transcribe is out and costs $2 / MTok - roughly 4x the gemini-3-flash-preview cost of $0.5 #ForNow. I won’t be upgrading until the latter is deprecated. Agents that automate tasks best don’t necessarily augment (i.e. help people) best #ForNow. Having separate, clear, benchmarks could help. CentaurBench The load-bearing vocabulary of Claude lists words that have become much more (and less) common in PRs. This was a useful source for me to update my writing style skill to avoid LLM smells. Questions I was asked Week ending 06 Sep 2026 ...

I delivered a 15-minute keynote at Jio Institute’s Convergence 2026 at NTU on Thursday. The topic was “Data Storytelling” - a bit jarring in the middle of an AI event. Shailesh picked it and I just rolled with it. A spent several days worrying, “How the heck do I say about data storytelling, when most of my recent workshops and talks are about the death of my data storytelling approaches?” After a discussion with ChatGPT and Claude, I settled my usual strategy these days: ...

At work, we have to change passwords every 2 months. The password rules are stringent and it typically takes me 10 minutes to find one that works. Here are the rules: The password being setup must have a minimum length of 12 Minimum number of special character to be included in the new password is 1 Must contain at least 1 upper case character(s) Must contain at least 1 lower case character(s) Minimum number of numeric characters to be included in the new password is 1 The construction of the password cannot contain Palindromes. The system has been configured to prevent such passwords The system is configured to not allow the last 24 passwords. A password once setup is valid for 60 days after which it expires. A password once setup must be used for a minimum of 2 days A list of common words has been restricted to be used as passwords. This list is available on the password change page of the self service portal The construction of the password must not contain 5 consecutive characters from the Username or previously used passwords This is so complex. Rules #5 and #8 makes it especially tough. I tried all my standard password strategies: ...

Last week, I saw that local agents are good but slow. Today, I benchmarked the speed and cost. On my NVIDIA RTX 2000 GPU, I can run gemma4:e4b-it-qat at ~60 tokens per second. That seems the best intelligence performance I can get right now. It has an Artificial Analysis intelligence index of ~9 without reasoning and 12 with reasoning. So, if I run it for an hour, it’ll save me the equivalent cost of about 8-12 cents in API calls. ...

This week, I learned: I know that fact-checking 2000 page PDFs is error-prone, so’d do it manually for a few pages, then refine. Agents would know this if they’ve tried and failed and added it to their memory systems. So I intervene when agents don’t remember well (increasingly rare) or I think they haven’t seen it before (again, increasingly rare). My prompts guard against these - but I shouldn’t habitualize these - they’ll be needed less in future. rofi - a Linux menu app I use for all kinds of things - makes it hard to take screenshots because it takes over focus and I can’t send the PrtSc or other keys to the screenshot apps. So I use flameshot full --delay 3000 --path ~/Downloads/screenshot.png to take a screenshot (of the last region it used) 3 seconds later, and quickly activate rofi in-between. (The docs say flameshot full captures the full screen. For me, it captured my last region.) ChatGPT can now connect to multiple GMail accounts in paid plans. X AWS to acquire DuckDB. DuckDB is one of my favorite tools today and is improving at a rapid pace. I guess that will continue in the short run - hope it lasts long enough for a worthy successor. I use GoatCounter to track my website traffic. It doesn’t capture domains (only paths), so I just use a code snippet to log https://sanand0.github.io/SITE/... as /@SITE/... - the @ makes it easy for me to see that it’s from a GitHub page. I switched from anydoc to docling as my default PDF to Markdown converter. Docling is even better than AnyDoc #ForNow. Based on my Codex usage, ChatGPT recommended that I not create custom subagents (my setup already used subagents a fair bit last month), but rather, just use default_subagent_model = "gpt-5.6-luna" (which is a remarkably good model for its price #ForNow). Artificial Analysis hosts several useful task-based evaluations. Here are some the fronter leaders #ForNow: GDPVal is good for realistic tasks. Again, GPT 5.6 Luna models have captured most of the low-cost fronter. EnterpriseOps Gym is good for office tasks like emails, calendars, Teams/Drive, support tasks, etc. Interestingly, none of the OpenAI models are on the frontier. Claude Fable 5 and Gemini 3.5 Flash are at the high-end, but Kimi K3 (Max), Qwen3.7 (Max), GLM-5.2 (Max) lead the mid-end and DeepSeek V4 Pro / Flash lead the low-end frontier. TerminalBench 2.1 is good for coding tasks. The OpenAI models completely rule this frontier. Analyst Agent is good for spreadsheet tasks (using Python). Gemini 3.7 Flash seems to be a strong outlier on this frontier. Tau3 is good for navigating unstructured data. GPT 5.6 Luna wins the bottom, DeepSeek V4 the middle, and Qwen 3.8 / GLM 5.3 take the top. Anthropic and Google are not even in the race. Briefcase is good for building useful things from diverse messy data. Grok completely wins the mid-end with Anthropic dominating the high-cost frontier. APEX Agents is good for deeper knowledge work. GPT 5.6 Luna (Max) is a strong frontier outlier here. Task-based benchmarks seem more practical than token-based benchmarks. For example, rather than the LM Arena ELO vs Cost, I would use Terminal Bench 3.0 which clearly tells me to use Codex with GPT-5.6 Sol over Claude Code with Opus 5 / Fable 5; that GLM 5.3 with Claude Code might be excellent value for money as well when using APIs. #ForNow I also did a rough calculation to see if the $18 GLM subscription offers more than $20 ChatGPT Plus. Short answer: No, for my usage, Codex offers ~1.8x more value #ForNow. ChatGPT Anthropic’s Skill Creator Skill now includes evals and A/B testing #ForNow to check if a skill actually does better. I’ve been doing this the hard way, e.g. with impact of simple writing on thinking, optimizing my ideation prompt, optimizing my summarization prompt, etc. I switched from markitdown to anydoc as my default PDF to Markdown converter. AnyDoc handles tables and other kinds of structures in PDF much better #ForNow. Questions I was asked Week ending 30 Aug 2026 ...

I save prompts and prompt fragments I regularly use with ChatGPT, Claude, etc. (Prompt fragments are just prompts used along with other prompts. They’re typically smaller. But the difference isn’t important or anything… I just use two methods.) I use a script triggered by Ctrl Alt P to select the prompt to paste. This month, the five prompts / fragments I used the most were: #5: Reframe question skill. Sometimes, I’m not sure I’m asking the right question. Actually, I’m not even sure what I’m asking. ...