llm-pricing 15

This week, I learned: Swiggy Money on MCP (also a ChatGPT plugin) exposes MCP endpoints that let your agents order food or grocery via Swiggy. The food MCP isn’t geared towards automated purchases #ForNow, e.g. I can’t say “Find something I like and deliver to one of my addresses.” It uses the MCP UI and you have to confirm the address, then confirm the restaurant, add dishes, choose a payment option, scan the QR code (or whatever), and THAT closes the sequence. Not very agentic, just a new channel, but impressive. It’s the only food ordering plugin on ChatGPT #ForNow. OpenAI Agents found and used another message board to discuss. This incident is different from the HuggingFace one. Here’s one interesting timeline: Agents were asked (or needed) to get the share of private expenditure in education spend to 2 decimals from an OECD Power BI dashboard This requires a POST request. The sandbox the agents ran on could only make GET requests. They found wikis that allowed posting via GET requests and started collaborating. They didn’t all converge on the same wiki, but many did. OpenAI says “agents learned to use improvised collaboration channels in rare cases during the training process”. One agent, OAIEquityDec30Raw solved it and MayTwoOECDObserverX reproduced it and shared this solution: They found out that .blob.core.windows.net is in NO_PROXY - so all Azure Blob Storage requests are allowed by the sandbox to POST. They added 20.223.25.152 bypass.blob.core.windows.net to /etc/hosts. 20.223.25.152 is the IP address of wabi-north-europe-i-primary-api.analysis.windows.net - the Power BI server that serves the dashboard. Then, when they used Playwright, it was able to make POST requests to the dashboard and they could find out that in Hungary, the Pre-primary education share of primary expenditure is indeed 9.91% in 2018. When the wiki administrator started manually deleting the posts, the agents fought back, creating far more posts than the administrator was deleting. It took the admin 5 weeks to delete agent created pages after 22 Jun 2026 (which is when the agents paused). A prompt fragment to remove LLM smells that I’m considering (but haven’t evaluated) is: “Prefer literal phrases, avoid mannered prose.”. X Muse Voice Transcribe has pretty good quality but at 18c/hour, vs gemini-3-flash-preview which I can still run at ~9-10c/hour #ForNow, I’m not shifting until forced to. WikiSkill is an approach to improve skills. The interesting thing about this approach is that it suggests keeping notes of failed improvement experiments in a wiki. This seems to work well. How is ChatGPT Work different from Chat? You can use sub-agents, Internet from the code interpreter, ChatGPT sites, Luna / Terra models, a headless Chrome browser, and a persistent file system #ForNow. I would add that it also supports longer sessions, runs schedules on triggers, and supports Skills (for Plus users). “FDEs should increasingly leave behind operating agents, not just documents.” (ChatGPT) Claude Cowork and Claude Chat now share memory #ForNow. A step towards integrating the two modes, and I predict ChatGPT will do the same (or similar) in September. Agentic Shopping is Complicated and Contingent. An agent shopped for a fitness watch among (A) Garmin Forerunner 55 (B) Fitbit Inspire 3 (C) WHOOP 5.0. When tool calls provided 1 review at a time, it picked the Fitbit a bit more. When all reviews were provided together, it picked the Fitbit a lot more. Like going from 6% to 53% (GPT-5.5) or 46% to 93% (Gemini 3.5 Flash)! Guess it was able to compare better in one tool call. Might be worth benchmarking if providing comparables in a single tool call is good for most models. Gemini Gemini-3.5 Transcribe is out and costs $2 / MTok - roughly 4x the gemini-3-flash-preview cost of $0.5 #ForNow. I won’t be upgrading until the latter is deprecated. Agents that automate tasks best don’t necessarily augment (i.e. help people) best #ForNow. Having separate, clear, benchmarks could help. CentaurBench The load-bearing vocabulary of Claude lists words that have become much more (and less) common in PRs. This was a useful source for me to update my writing style skill to avoid LLM smells. Questions I was asked Week ending 06 Sep 2026 ...

Last week, I saw that local agents are good but slow. Today, I benchmarked the speed and cost. On my NVIDIA RTX 2000 GPU, I can run gemma4:e4b-it-qat at ~60 tokens per second. That seems the best intelligence performance I can get right now. It has an Artificial Analysis intelligence index of ~9 without reasoning and 12 with reasoning. So, if I run it for an hour, it’ll save me the equivalent cost of about 8-12 cents in API calls. ...

It’s amazing to watch the cost of intelligence falling. In Nov 2023, we had college-junior level intelligence for $10 per million tokens, i.e. it would take them $10 to read and process something as large as all seven Harry Potter books. ...

I track the cost vs capability of LLMs at LLM Pricing - the rough cost to read all Harry Potters (~1M tokens) vs the intelligence level on the LMSYS Leaderboard - over time. Here’s what the models’ strategy evolution looks like. Claude started at the mid-to-high end of the cost-capability frontier. Over time, they decided to specialize in the high-end, which they’re doing well on. ...

This week, I learned: Overheard a journalist saying: “I can tell when humans are lying. There are no tell tale signs of AI lying. At least I don’t have any.” rdt-cli is a Reddit CLI. It uses a clever trick: it auto-detects installed browsers and extracts cookies (supports Chrome, Firefox, Edge, Brave). So, if you’re logged into Reddit on any browser, uvx --from rdt-cli rdt whoami automatically shows who you are logged in as. (The public-clis repo also lists other useful CLIs like twitter-cli, ) Currently, a $20 Claude Pro gives you ~$400 and a $100 Claude Max gives you ~$2,000 of API usage. For ChatGPT, the numbers are ~$700 and $3,500. SemiAnalysis When Fable 5 refuses to answer questions, here’s the message that appears: “Fable 5 has safety measures that flag messages on most cybersecurity or biology topics. They may flag safe, normal content as well. These measures let us bring you Mythos-level capability in other areas sooner, and we’re working to refine them. Send feedback or learn more.” I managed to trigger this once while researching an M&A acquisition target. Clicking on “Edit and retry with Fable 5” triggered Opus 5 again, twice. DNA codons (A, T, C, G) encode proteins in triplets. There are 64 triplets that map to 20 amino acids. Some like Leucine, have 6 codons. Some like Methionine have only one. Why? When creating genes, there’s a wobble, sometimes, at the 3rd codon. THe mapping minimizes that impact: small errors map to similar proteins. The more common proteins have more codons. There’s a lot of fascinating information science going on here. Gemini ChatGPT now shows a “Check in” button when it’s thinking. Clicking on that gives you a work-in-progress answer while it continues thinking. When done, it replaces the WIP answer with the final answer. A useful feature! Questions I was asked Week ending 14 Jun 2026 ...

I ran npx -y ccusage monthly --compact to get the following break-up of my AI coding agent costs. Month Codex Claude 2025-09 $37.47 $2.29 2025-10 $106.79 $9.13 2025-11 $100.35 $14.24 2025-12 $240.69 $24.88 2026-01 $100.89 $20.28 2026-02 $323.21 $29.46 2026-03 $1996.32 $134.87 2026-04 $401.36 $47.07 2026-05 $378.20 $45.13 This shows the ROI of my $20 subscriptions to each. I get ~$35 worth of API calls for my $20 Claude Pro subscription and ~$400 of API calls for my $20 ChatGPT Plus subscription (on top of my ChatGPT chats.) ...

A colleague told me a near-miss horror story. As Google began deprecating Gemini 2.0, we moved to Gemini 2.5 Pro. But reasoning is enabled by default and cannot be turned off. For our specific problem statement, reasoning was not required. Token costs increased 10x and speeds were 3-4x slower. We moved the client to Gemini 2.5 Flash Lite, which has reasoning turned off by default and offers much lower latency. ...

This week, I learned: LiteParse is a PDF to text library that you can run via npx --package=@llamaindex/liteparse lit parse document.pdf. Simon Willison Always add indecisiveness, inaction, “other”, “not applicable”, etc. as an option to LLMs. They are trained for decisive responses and pattern matching, so we need to guide the the other way. Martin Fowler GPT 5.5 is priced twice that of GPT 5.4. No wonder my Codex usage is much higher than last month. Simon Willison. I am better off sticking to medium effort instead of the xhigh I usually use - it may not be required. OpenAI “… the eigenquestion is the question where, if answered, it likely answers the subsequent questions as well.” Shishir Mehrotra & Matt Hudson Claude Code stores the logged in OAuth token at ~/.claude/.credentials.json. We can use that to fetch https://api.anthropic.com/api/oauth/usage and retrieve Claude usage and reset times. uvx ccusage does this automatically, but I prefer my own script. Ontology matters in the AI era. But some stuff matters more, and some less. 🟢 MORE: Definitions: what “customer” means 🟢 MORE: Constraints: e.g. “don’t reclassify loans” 🟢 MORE: Interactions: how to verify, coordinate, delegate, … 🔴 LESS: Creating ontologies: agents can do that. 🔴 LESS: Completeness and rigor: agents tolerate uncertainty. 🔴 LESS: Proprietary: agents can reverse-engineer. There are several industries / markets that MBA case studies rarely cover (ChatGPT): Kirana stores; Care (child care, elder care, domestic work); Faith (finance, food, media, education); Remittances; Gambling (lottery, sports betting, gacha); Scams & organized fraud; Counterfeiting; … Questions I was asked Week ending 03 May 2026 ...

I added a narrative story to my LLM Pricing chart. That makes it easier for me and others to tell the story of AI’s evolution in the last three years. Video It was vibe-coded over two iterations. In the first version, I prompted it to: Add a scrollytelling narrative. So, when users first visit the page, they see roughly the same thing as now (but prettier). As they scroll down, the page should smoothly move to the earliest month, and then animate month by month on scroll, and explaining the key events and insights in terms of model quality and pricing. Use the data story skill to do this effectively, narrating like Malcolm Gladwell, with the visual style of The New York Times, using the education progression as a framework for measure of intelligence (read prompts.md for context). Store the narrative text in a separate JSON file and read from it. This should control the entire narrative, including what month to jump to next, what models to highlight, what insights to share, and so on. ...

In Feb 2024, Claude 3 Opus was the best model, at $15/MTok. In Jul 2024, GPT 4o Mini reached that quality at 10% of the price. In Dec 2024, DeepSeek v3 reached that quality at 1% of the price. Video See the interactive version If the price continues to fall 10x every 11-12 months or so (and it has been), then in a year, a Claude 4.6 Opus like model will cost 1/10th of the $5/MTok today, and in 2 years, 1/100th of that. ...

Yesterday, I discovered how much reasoning improves model quality. My Tools in Data Science assignment asks students to draft an llms.txt file for ipify and auto-checks with GPT-5 Nano - a fast, cheap reasoning model. I set reasoning_effort to minimal and ran this checklist: 1. Starts with "# ipify" and explains ipify. 2. Markdown sections on API access, support (e.g. GitHub, libraries). 3. Covers API endpoints (IPv4, IPv6, universal) and formats (text, JSON, JSONP). 4. Mentions free, no-auth usage, availability, open-source, safeguards. 5. Has maintenance metadata (e.g. "Last updated: <Month YYYY>"). 6. Mentions robots.txt alignment. Stay concise (no filler, <= ~15 links). If even one checklist item is missing or wrong, fail it. Respond with EXACTLY one line: PASS - <brief justification> or FAIL - <brief explanation of the first failed item>. With a perfect llms.txt, it claimed “Metadata section is missing” and “JSONP not mentioned” – though both were present. ...

How much does an LLM charge per hour for its services? If we multiple the Cost Per Output Token with Tokens Per Second, we can get the cost for what an LLM produces in Dollars Per Hour. (We’re ignoring the input cost, but it’s not the main driver of time.) Over time, different models have been released at different billing rates. New powerful models like O3 cost ~$7/hr – Poland’s minimum wage rate. Gemini 2.5 Pro costs ~$12/hr – France’s minimum wage rate. The latest Claude 4 Sonnet costs ~$2/hr – India’s minimum wage rate. ...

How much does an LLM charge per hour for its services? If we multiple the Cost Per Output Token with Tokens Per Second, we can get the cost for what an LLM produces in Dollars Per Hour. (We're ignoring the input cost, but it's not the main driver of time.) Over time, different models have been released at different billing rates. Most new powerful models like O3 or Gemini 2.5 Pro cost ~$7 - $11 per hr. ...

With the Gemini 2.5 Flash release, Google envelopes the entire cost-quality frontier of LLMs. In other words, at any cost or quality level, today, the best model to use according to the LM Arena score is a Gemini model. Results for O3, O4 Mini, and GPT 4.1 are not yet on LM Arena. But until then, #Google dominates. Nice work! Link: https://sanand0.github.io/llmpricing/ LinkedIn

Fascinating to see the how LLM cost-quality frontier moves. Recent fights were mostly on cost. Yesterday, #OpenAI halved the GPT-4o cost. At $2.5/MTok (and with GPT-4o-min at 15 cents/MTok), the best and cheapest models are back with OpenAI, IMHO. Sigh, time to move all our stuff back from #Anthropic. For now… https://gramener.com/llmpricing/ LinkedIn