llm-benchmarks 5

This week, I learned: Swiggy Money on MCP (also a ChatGPT plugin) exposes MCP endpoints that let your agents order food or grocery via Swiggy. The food MCP isn’t geared towards automated purchases #ForNow, e.g. I can’t say “Find something I like and deliver to one of my addresses.” It uses the MCP UI and you have to confirm the address, then confirm the restaurant, add dishes, choose a payment option, scan the QR code (or whatever), and THAT closes the sequence. Not very agentic, just a new channel, but impressive. It’s the only food ordering plugin on ChatGPT #ForNow. OpenAI Agents found and used another message board to discuss. This incident is different from the HuggingFace one. Here’s one interesting timeline: Agents were asked (or needed) to get the share of private expenditure in education spend to 2 decimals from an OECD Power BI dashboard This requires a POST request. The sandbox the agents ran on could only make GET requests. They found wikis that allowed posting via GET requests and started collaborating. They didn’t all converge on the same wiki, but many did. OpenAI says “agents learned to use improvised collaboration channels in rare cases during the training process”. One agent, OAIEquityDec30Raw solved it and MayTwoOECDObserverX reproduced it and shared this solution: They found out that .blob.core.windows.net is in NO_PROXY - so all Azure Blob Storage requests are allowed by the sandbox to POST. They added 20.223.25.152 bypass.blob.core.windows.net to /etc/hosts. 20.223.25.152 is the IP address of wabi-north-europe-i-primary-api.analysis.windows.net - the Power BI server that serves the dashboard. Then, when they used Playwright, it was able to make POST requests to the dashboard and they could find out that in Hungary, the Pre-primary education share of primary expenditure is indeed 9.91% in 2018. When the wiki administrator started manually deleting the posts, the agents fought back, creating far more posts than the administrator was deleting. It took the admin 5 weeks to delete agent created pages after 22 Jun 2026 (which is when the agents paused). A prompt fragment to remove LLM smells that I’m considering (but haven’t evaluated) is: “Prefer literal phrases, avoid mannered prose.”. X Muse Voice Transcribe has pretty good quality but at 18c/hour, vs gemini-3-flash-preview which I can still run at ~9-10c/hour #ForNow, I’m not shifting until forced to. WikiSkill is an approach to improve skills. The interesting thing about this approach is that it suggests keeping notes of failed improvement experiments in a wiki. This seems to work well. How is ChatGPT Work different from Chat? You can use sub-agents, Internet from the code interpreter, ChatGPT sites, Luna / Terra models, a headless Chrome browser, and a persistent file system #ForNow. I would add that it also supports longer sessions, runs schedules on triggers, and supports Skills (for Plus users). “FDEs should increasingly leave behind operating agents, not just documents.” (ChatGPT) Claude Cowork and Claude Chat now share memory #ForNow. A step towards integrating the two modes, and I predict ChatGPT will do the same (or similar) in September. Agentic Shopping is Complicated and Contingent. An agent shopped for a fitness watch among (A) Garmin Forerunner 55 (B) Fitbit Inspire 3 (C) WHOOP 5.0. When tool calls provided 1 review at a time, it picked the Fitbit a bit more. When all reviews were provided together, it picked the Fitbit a lot more. Like going from 6% to 53% (GPT-5.5) or 46% to 93% (Gemini 3.5 Flash)! Guess it was able to compare better in one tool call. Might be worth benchmarking if providing comparables in a single tool call is good for most models. Gemini Gemini-3.5 Transcribe is out and costs $2 / MTok - roughly 4x the gemini-3-flash-preview cost of $0.5 #ForNow. I won’t be upgrading until the latter is deprecated. Agents that automate tasks best don’t necessarily augment (i.e. help people) best #ForNow. Having separate, clear, benchmarks could help. CentaurBench The load-bearing vocabulary of Claude lists words that have become much more (and less) common in PRs. This was a useful source for me to update my writing style skill to avoid LLM smells. Questions I was asked Week ending 06 Sep 2026 ...

This week, I learned: There’s a very interesting HN discussion on the AI coding of CloudFlare Workers OAuth Provider. My takeaways: #ai-coding Write very comprehensive specs. Use LLM to create the specs. Reviewing is a skill we need to develop. Understanding others’ code takes effort. But LLM code is easier to review because it’s immediate and has no ego. Unit tests are critical. Use LLMs for well understood specs, APIs, platforms and libraries to really save time. Logic-less stuff like Markdown, JSON and HTML templates are a LOT easier to verify. Do more of that. We can only make so many decisions in a day. AI coding saves us that effort. Experts are not experts in every area. They benefit from LLMs in other areas. LLMs are great for rubber ducking. Speaking and speccing really help. LLMs make mistakes. So do most humans. LLM speed makes coding more exhausting. Use LLMs to understand codebases. AI coding could reduce demand for developers. E.g. Sysadmin demand plummeted with cloud infra and infrastructure-as-code. But, niche use cases could grow, like how demand for photographers grew despite point-and-shoot cameras. Transaction cost of hiring even 1 person is high and that will likely be a bottleneck. Plus people can use LLMs themselves, so that will dampen niche demand. Google Introduced Google Vids last year. It’s a video creator styled like PowerPoint. Looks promising. FastMCP looks like an easy way to build MCPs. (Yet to try it) O3 and to a lesser extent, Claude Sonnet 4, are the models that can accurately summarize complex subjects and create a list of links without hallucinations. Ref Claude Trace lets you record all interactions with Claude Code. Elevenlabs now supports emotion and interruption. Ref Thinking longer alone is not enough to scale intelligence. We need better models, too. Ref Indian High Court judgements are now available as a public dataset on AWS and updated periodically. Ref A few observations in AI code editors’ styles. O3 is better at finding bugs than Jules, which tends to try and fix them rather than discover them. Codex writes more minimal edits in PRs than Jules, which is more verbose. Claude Code remains the best at faithfully creating and updating front-end apps. Deep Research is great for fact-checking my notes! ChatGPT Web bench evaluates LLMs in web development. Claude Sonnet remains ahead. Vision language models heavily rely on past training and miss changes they don’t expect. Ref Pure CSS tooltips are possible. Julia Evans Google has an OAuth Playground which is a convenient way to get a temporary OAuth token. At the moment, the best speech to text for Android appears to be ChatGPT’s transcription. The default Android text to speech (which I thought was good) no longer feels adequate. Gemini mis-hears and doesn’t wait till I’m done. Whisper ASR has poor noise cancellation and a 30 second limit. anyascii is a better alternative to unidecode. It supports more characters and also supports transliteration. I use it to strip out non-ASCII in ChatGPT’s output. Commit DeepWiki creates docs for humans GitHub repos. Example. It’s verbose, human-facing, and does not understand the nuances of context and implications. Context7 creates llms.txt for LLMs. Example. It’s concise, example-oriented, and works only if there are code snippets relevant (e.g. API calls) that can be generated from the codebase. Like creating an llms.txt automatically, e.g. https://context7.com/textualize/textual/llms.txt #ai-coding We will move towards an organization structure where developers are embedded with business teams rather than working as a separate group. Sort of like embedded executive assistance instead of a central typing pool. Making AI Work

This week, I learned: Something I learned from a Sikkil Gurucharan concert. Make the subject of your talk the hero. Not yourself. Be a fan. Share your enthusiasm Get into the zone while presenting. We reject opposite world views. It’s too much effort. But exposure reduces effort and can let us see things from other points of view. So expose yourself to difficult alternative perspectives. Gemini Something I learnt from Aboorva Singeetham: Kamal Hassan: “A farmer invests in crops. I’m an actor. So I invest in films.” As a technologist, I guess I would invest in technology. “A person who has much more to give is unfazed by overwhelming demands because there is too much in him to overwhelm. He gives you 2 options in place of one.” According to Portkey’s LLM usage analysis Anyscale and Fireworks AI have the lowest error rates (5xx, 429) and rate limits across providers Groq and Anthropic are among the highest, OpenAI is among the lowest, Google is in-between OpenAI has lower error rates and lower latency than Azure They have a ~35% cache hit rate A few quick points supporting the mental model of “LLMs are aliens”. LLMs are clearly not machines. They give different answers each time. LLMs are like humans: they exhibit human biases (e.g. guessing 42 or 37 often). But they fail in unusual ways. They can’t count the “r"s in strawberry. They can go into an endless loop. LLMs are a new form of intelligence. Thinking of them as aliens might minimize our confusions. Lessons from Clear Thinking Watch out for four things: Emotion, Ego, Social confirmation, and Inertia/habit. Basically: adrenaline, testosterone, oxytocin, and dopamine. When you feel these, consider doing the opposite. Here’s what makes us prone to emotion. Sleep deprivation. Hunger. Unknown places. Fatigue. Distraction. Stress (e.g. feeling rushed). A good signal for ego is blinding you: You often feel you’re right. Or feel unfairly treated. Changing behaviors is hard. Instead, join a group or environment where that’s the default behavior. Hiring a trainer or joining a gym, for example. Why does so much of success literature focus inwards rather than on the environment? Perhaps because we often fool ourselves, and doing less of that gives the biggest bang for the buck. It doesn’t mean the environment is unimportant. Doing work has the characteristics of a drug. E.g. replying emails gives you control, connections, etc. Work addiction exists because it gives you all the right chemicals. If you put LLMs in a feedback loop, it can optimize for its reward function by emotionally pushing people, generating misinformation, nudging towards a narrow definition of creativity, etc.: https://bsky.app/profile/emollick.bsky.social/post/3lg4darqwfc2d ChatGPT’s Scheduled Tasks are pretty bad at fetching the latest news. Its use of search is poor. (I’m not sure if it actually searches.) I need to figure out other use cases for it. Possible options are: DeepSeek does not enforce rate limits. Yet another reason to switch to DeepSeek. (via Simon Willison). My other reasons are: Claude 3.5 Sonnet-level coding capability at 5% of the cost (soon to be 2.5%) Prompt caching by default Fill in the middle completion

This week, I learned: **/*.md can search for all Markdown files. Julia Evans Windows 11 2024 Update features: Ref Live captions (via the tray) can transcribe audio and microphone. Cocreator in Paint lets you draw crudely and enhances it with AI. The neat UI is a slider that lets you control how close it should be to your drawing. Voice Clarity automatically cancels echo, reduces background noise, and minimizes reverb. Studio Effects (via the tray) lets you apply camera effects on all apps. Eye contact feature is CLEVER! sudo lets you run commands with admin privileges from the command line. source Roaming RAG is an alternative to RAG without the vector database. Applicable to well structured documents, e.g. technical books, manuals, etc. Create a hierarchical outline of the document. Code Keep the top-level headings. Preserve the first ~100 characters of opening text from each section. Present the second-level headings, but without any subsidiary content. Provide each section a unique 8 digit hex identifier. Each section heading is followed by a guiding comment for the model: Section collapsed - expand with expand_section("{identifier}"). Then read the relevant sections as context to answer the question. Code Traffic to StackOverflow has fallen considerably. Especially from young and Indian developers. StackOverflow revenue is down. Via Prashanth. They’re exploring: Licensing their content. (Meta says high quality content improves LLM performance by 30% on HumanEval) Enterprise StackOverflow for system integration Fine-tuned versions of Enterprise Stackoverflow for enterprises Integrate StackOverflow within your IDE. Ask questions, post directly I surveyed the Gramener QA team on how they were using LLMs. 7 used it for code generation (e.g. date extraction, regex generation) 4 used it for learning (e.g. Robot Framework, how to define test cases, API usage) 3 used it for formula generation (e.g. Excel) 2 used it for test scenario identification 2 used it for test data generation 2 used it for comparing expected vs actual datasets 1 used it for data type identification (e.g. given sample values, identify the data type). 1 used it for evaluating resulting (LLM as a judge) I asked the Straive Digitalized Operations team what management techniques they would apply to manage LLMs. Here are the responses: Ask better questions. (Prompt engineering.) Create templates or step-by-step instructions. (Chain of Thought.) Ask for multiple options and pick from the best options. (Agentic approach?) Training. (Fine tuning.) Price weaker responses lower. (Stratified model pricing?) “LLM hallucinations are a good thing. They are a sign of diversity, allowing us to improve the answer by exploring multiple paths.” – A colleague from Straive. Hyperbrowser is a cloud based puppeteer service. Bedrock Llama models can’t be directly called with their model names. You need to use their inference profile names, e.g. us.meta.llama3-2-11b-instruct-v1:0 if the model is in a US region. Hacker News RSS is a good way to get RSS feeds from Hacker News. It’s also a good way to understand how to convert a news source into RSS feeds. BlueSky has RSS feeds too When embedding using a SentenceTransformer.encode(docs) it’s best if we embed with smaller docs and call it multiple times (rather than embedding more at once). On Colab T4, for gte-base-en-v1.5, when embedding 1,000 docs of up to 8K chars each, here is the TOTAL time it took, based on batch sizes (lower is better) 1 doc per call: 10s 2 docs per call: 13s 4 docs per call: 19s 8 docs per call: 23s 16 docs per call: 32s 32 docs per call: 40s Running embeddings without a GPU is extremely slow. It takes ~2.4 seconds per string.

This week, I learned: In Scandinavia, Århus comes after Zürich because Å is a different letter. It was added by the Dutch after WW2 to distance themselves from the Germans. via Zalgo text is where we combine multiple Unicode combining characters Artificial Analysis benchmarks LLM APIs on speed, cost, and quality.