coding-agents on S Anand

Let AI take your exams

Fri, 12 Jun 2026 08:10:56 +0530

At 2 pm IST today (Fri 12 Jun 2026), I conducted a workshop at Paradox, IITM - at DOMS 101.

My core message is: “AI can solve exams and help you learn. Delegate what AI can do. Learn what AI can’t do instead.”

My talks page for “Let AI take your exams” includes:

The full story + transcript + audio
How Codex solved a real exam, live
My collection of AI-learning techniques - which was not covered in the workshop, but is a useful reference

Here are the takeaways from the workshop:

AI is more capable than you think — and getting smarter. Recalibrate constantly what it can and can’t do. Note down what it can’t, because that is precisely where your value lives.
Delegate first; learn the rest. Give everything to AI. Focus your learning on what it can’t yet do — that’s where the value will be. It’s a moving filter; revisit it every quarter.
Always use the best model, turned up high. Reserve “fast and cheap” for the ~5% of moments you need a quick answer. And remember students get serious AI free via the GitHub Student Pack and Gemini.
Make the AI ask you for context. “If you need more information, ask me.” You don’t have to know what context it needs — push that burden back to the machine.
Beat hallucination with a maker-checker. Two independent models that must agree cut errors from 14% to under 4%. Tell the checker to “find the errors,” not to grade.
Loop with feedback in verifiable environments. Point an agent at an exam, a codebase, anything that scores itself — let it try, submit, read the result, retry. This is the most powerful technique AI has.
Calibrate, don’t just trust. Practise predicting whether AI will get something right — even on topics you don’t know. Watch for base-rate traps and familiar problems with one changed premise.
Be lazy, productively. Don’t read AI’s 20-page output — train it to give you five words. Working well with AI is a management skill.
Learn from peers. Multiple people trying things is how you discover what works. Non-transactional relationships are the rising currency of the AI era.
Apply the scientific method to everything. Form a hypothesis, hunt for evidence, try to falsify yourself. And when a system blocks you unfairly — hack it, then publish what you learned.

Here are the takeaways from how Codex solved the exam:

An agent operates the environment; a chatbot answers the question. Codex read source, ran code, clicked Check, and looped on feedback. That’s why it beat copy-paste — the exam was full of affordances a chatbot can’t touch.
Verifiable environments favour AI. The more checkable the exam — validators, error strings, downloadable files, a live Check button — the more it helped the agent, not the student. “AI-proof” and “feedback-rich” are opposites.
Most failures were wording, not reasoning — and the source fixed them. The fix for a brittle validator was to read the validator. Pass the error message back to the agent and it converges.
Don’t guess where attempts are limited. The network game punished early guesses. The recoverable mistakes had feedback; the costly ones didn’t. Triage cheap-and-certain first.
The gap between 9 and 10 was a credential, not a brain. Same model, same skill. Anand’s missing mark was an invalid token. In the AI era, “can it?” often means “does it have the keys?”
It cost about a coffee. ~$2–3 of tokens for the whole exam, ~96% cached. The real cost is the human judgment to know when the agent is plausibly, confidently wrong.

Original announcement

You can join online at https://meet.google.com/cpt-faee-ucx and ask questions on chat.

Agenda:

You’ve been told AI can pass your exams. But what happens when you actually watch it try — live, on your questions, in real time?

This workshop starts with a collective experiment: we ask coding agents to solve real exams (including IITM exams) and see how it solves them.

What follows isn’t a tutorial on prompting — it’s an autopsy that reveals what your exams are actually testing, where AI confidently hallucinates, and what that means for what’s worth learning.

You’ll leave with a reframed understanding of your degree (the goal isn’t answers, it’s the ability to catch wrong ones) and a concrete study rituals that uses AI as a Socratic sparring partner rather than an answer machine.

Come with a question you got wrong recently — it’s going to be useful.

Real agenda: An ask-me-anything session plus real-life experiments.

AI Coding Agent Subscription ROI

Sat, 30 May 2026 23:19:34 +0800

I ran npx -y ccusage monthly --compact to get the following break-up of my AI coding agent costs.

Month	Codex	Claude
2025-09	$37.47	$2.29
2025-10	$106.79	$9.13
2025-11	$100.35	$14.24
2025-12	$240.69	$24.88
2026-01	$100.89	$20.28
2026-02	$323.21	$29.46
2026-03	$1996.32	$134.87
2026-04	$401.36	$47.07
2026-05	$378.20	$45.13

This shows the ROI of my $20 subscriptions to each. I get ~$35 worth of API calls for my $20 Claude Pro subscription and ~$400 of API calls for my $20 ChatGPT Plus subscription (on top of my ChatGPT chats.)

I end up using Codex a lot more - partly because it’s a bit more diligent, but mostly because it’s a lot cheaper.

Clearly, subscriptions are good deal for individuals. Codex, especially.

This may not be true for corporates. Simon Willison says that Anthropic and OpenAI both changed enterprise pricing to align with token prices. That means the cost of enterprise AI security is ~2-20 times their token budget - which is growing rapidly.

BTW, my moment of AI psychosis was in March 2026. The coding agents had increased their limits and I was tokenmaxxing. I’m far from that limit today, but the symptoms linger.

14 Jun 2026: SemiAnalysis tested and found that a $20 Claude Pro gives you ~$400 and a $100 Claude Max gives you ~$2,000 of API usage. For ChatGPT, the numbers are ~$700 and $3,500.

Correcting instruction debt

Mon, 25 May 2026 16:30:02 +0800

Here’s another AI-generated post, with Anand editor notes. But I’ve also added my own version of the post below.

I told my “find a free calendar slot” script to “Avoid weekends and holidays”. Wednesday vanished. Turns out it’s a Singapore holiday (Anand: It’s Eid al-Adha), — irrelevant for the people I was meeting in other zones. I’d debugged my own helpful rule. (Anand: What? What does “debugged my own helpful rule” even mean?)

Annoying. But revealing. I went to ChatGPT — not to fix the script, but to think: (Anand: True. I had no clue what to do.)

Could you maybe interview me to figure out what direction I might want to take this train of thought in…? Just ask me two or three questions.

(The whole point was to not take on more work myself.)

Two questions in, it named it: instruction debt. (Anand: which is such a cool term that I’ll keep it.)

Not “bad instructions,” because the original instruction was reasonable. The debt is created when a rule that once reduced cognitive load later creates invisible work, missed options, brittle behavior, or debugging cost.

That hit. The script obeyed too literally. I got no warning. Worst of all, I’d scored a self-goal — given my future self an instruction that would bother me, while believing I was being helpful.

I asked it to research further — and to mine my own agent logs as evidence. (Local MCP runs bash; ChatGPT can read ~/.codex, ~/.claude, ~/.copilot and run ~/code/scripts/agentlog.py directly.) It came back with a taxonomy. I asked it to stress-test against more correction turns and discard what didn’t survive. (Anand: Basically, I said, analyze my logs.)

It did. The robust categories, each grounded in an actual correction I’d made:

Objective framing — “don’t base teachability on scores… base it on the pattern of errors.” Wrong proxy. (Anand: Oh, yeah, I was trying to find patterns of errors in student submissions.)
Evidence/modeling — Ticketmaster classifier overfit on venue_name. Predictive, not causal. (Anand: True. Stupid model said, “tickets in this stadium sell more” as if it were actionable.)
Constraint semantics — the Singapore holiday. Hard filter where a warning would do.
State/action — Darwinbox: “Click Clockin” clocked me out. No pre/post-state check. (Anand: The button said “Clock in OR out”. I was clocked in. It clicked, thinking that’ll clock me in, without seeing that the button was already pressed.)
Representation/path — blog migration: “ALL LINKS relative” broke nested URLs. (Anand: Yeah, relative links in my blog have been problematic for 20 years.)
Validation — OCBC PDF: row balances passed, totals failed by SGD 6.9M. (_Anand: I’m nowhere near this rich. Codex just messed up badly.)

ChatGPT’s own self-critique was the best part:

“Lack of carefulness” should not be a category. It is not actionable. (_Anand: No idea what this means!)

Then the pivot. It proposed a 60-line “Operating Contract” for my AGENTS.md. I pushed back:

The operating contract is WAY too long. I was thinking 1 line, not 60… fast and frugal heuristics that cover the majority of the scenarios, rather than hard-coding everything, is what we’re suggesting coding agents do in the first place.

It came back with three lines. I pasted them in verbatim:

For non-trivial tasks, define the user-visible invariant: "done means ___"; verify that invariant before claiming success.
Treat constraints as soft preferences unless safety, privacy, data loss, credentials, or the current request makes them hard; surface any constraint that filters, skips, blocks, or deletes.
Prefer simple, rerunnable changes: inspect real inputs/state first, use existing tools/libs, log counts/examples, and call out uncertainty.

Line 1 catches OCBC and clock-in. Line 2 catches the Singapore holiday. Line 3 catches the toil.

Don’t add the rule to your head. Add it to the file the agent already reads. (Anand: Oh, so totally true!)

Actually, the first half of the above AI-generated post didn’t really resonate with me. So let me explain in my own words what I did.

I found that, for some reason, this Wednesday never appears when I ask for find a free calendar slot.
I asked Codex, “Why on earth is this happening?” It said, because you told me to exclude holidays.
That got me thinking, where am I giving instructions that shoot me in the foot? And ChatGPT did a long, detailed analysis of my coding agent logs and came up with a bunch of examples and categorization.
I didn’t bother reading it. I told it in Henry Kissinger style: can you do better?
I didn’t bother reading it again. I told it, “Just tell me what to put into AGENTS.md”. I don’t want to do the work every time. YOU do the work. Automate it!
It gave me 60 lines. I said, “What rubbish! I can’t review 60. Just 3, max.”
I copied that into AGENTS.md.

For non-trivial tasks, define the user-visible invariant: “done means ___”; verify that invariant before claiming success.

Treat constraints as soft preferences unless safety, privacy, data loss, credentials, or the current request makes them hard; surface any constraint that filters, skips, blocks, or deletes.

Prefer simple, rerunnable changes: inspect real inputs/state first, use existing tools/libs, log counts/examples, and call out uncertainty.

The first makes total sense. Define “done”.
The second makes some sense - that’s exactly what I did wrong with the calendar.
The third is supposed to “handle my recurring style” - and kind of makes sense, so I’ll let it be.

AI advice for teams

Sat, 25 Apr 2026 12:17:03 -0400

I updated my AI Advice page by:

Transcribing my calls in the last 2 months (Gemini 3.1 Pro, “Transcribe this call recording…”)
Extracting AI advice (Gemini 3 Flash, “Summarize ALL AI-related advice … into 1-sentence bullets”)
Asking Claude, ChatGPT, and Gemini to document what’s new / changed.

I added this request:

But, and this is IMPORTANT, analyze my original writing style, write it exactly in that style, and then verify to make sure it follows the same style (correcting where required.)

This worked surprisingly well with Claude 4.6 Sonnet (adaptive). I used the resulting ai-advice.md almost verbatim.

Here’s the full list of changes it suggested:

The biggest structural update: ai-advice.md is still written as “how individuals use AI better.” Your actual advice has evolved into “how people, teams, and organizations operate in the agent era.” The whole document should eventually be reorganized to reflect this.

INSERT

Build an AI workspace, not just a chat (Very high frequency — 10+ docs) Every serious AI project needs a project folder containing:
- AGENTS.md — folder-specific instructions the agent reads on startup
- prompts.md — all prompts version-controlled as source code
- skills/ — encapsulated successful workflows (see #2)
- Git repository with commits at every checkpoint
- Test fixtures, synthetic datasets, logs, outputs
Treat prompts as the real IP. Code is disposable; prompts, tests, and skills are assets.
Encapsulate successful workflows into reusable skills (Very high frequency) Once an agent succeeds at a task three times, encapsulate it: the prompt, tools used, edge cases, constraints, validation tests. Store in a skill.md file. Skills are the new software libraries — they make workflows deterministically repeatable without re-explaining everything. Use agents to build these skills by asking them to summarize what they learned.
Run coding agents safely: Git + Docker (Very high frequency) Always: (a) work inside a Git repository and instruct the agent to commit as it goes — git checkout is your undo button, (b) run agents inside Docker containers so they cannot touch your actual files, (c) use “YOLO mode” (skip permission prompts) only inside isolated containers. These aren’t optional for anything beyond throwaway prototypes.
“LLMs hallucinate, but code doesn’t” — use code as the truth engine (Very high frequency) Broaden “have it write code to process numbers” significantly. The mantra is: wherever correctness matters, make the AI produce executable code or logic rather than natural language answers. Code either works or fails — it’s binary and auditable. Use domain-specific languages (Prolog-like rule trees, schema validators, policy-as-code) for logic-heavy tasks. This is the primary mechanism for eliminating hallucinations in production.
Build verification into the workflow, not after it (Very high frequency) Verification should be engineered as a product feature, not added as a post-hoc check. Every output should expose: source citations linked to snippets, confidence levels, what’s unverifiable, disagreement signals, and audit logs. Use model disagreement as a routing signal — when models disagree, send to human review; when they agree, lower review priority. Build golden sets to measure actual accuracy on your specific task.
Use AI for exception triage, not blanket automation (High frequency) Let AI classify outputs as red/yellow/green: green = automate fully, yellow = flag for review, red = human required. This is more mature than “80-90% AI, human for last mile.” It says exactly where the human loop belongs, and it scales: automation handles routine volume while humans focus only on high-stakes exceptions.
Use synthetic data deliberately (High frequency) Not just “realistic fake data for prototyping” — generate hypothesis-driven synthetic data that embeds specific behavioral patterns, edge cases, and known failure modes you expect in production. This lets you stress-test before real data arrives, without compliance concerns, and at whatever messiness level you choose.
Treat demos as imagination accelerators (High frequency) Demos are not just proof-of-concept — they are the fastest way to expand what stakeholders think is possible. Use “Hollywood set” demos: working outputs, simulated backends, precomputed workflows, client-specific synthetic data. Only demo live if the task completes in under 10 minutes. Simulate or precompute slow, expensive, or credential-heavy workflows. Show the output first; defend the architecture only if asked.
Maintain a living model radar — don’t freeze model advice (High frequency) Specific model recommendations go stale within months. The durable advice: continuously blind-test frontier models on your exact task, maintain a benchmark set, and route by capability. Current pattern: Claude for coding/aesthetic/style/writing; ChatGPT for rigorous analysis/financial modeling/extended thinking; Gemini for Google Workspace/research/video/speed. But measure this; don’t assume it. Additionally: use LiteLLM or Portkey as open-source gateways for organizational cost observability across models.
The Jevons Paradox applies to knowledge work (High frequency in strategic contexts) AI making cognitive tasks cheaper will increase total demand for cognitive work, not reduce it. Human roles shift from execution to verification and judgment — but there’s a talent crunch coming for verification roles. Hire now for people who can check, certify, and take accountability for AI output.
Use games to teach AI, not slide decks (High frequency) Replace passive L&D with Capture the Flag challenges, treasure hunts, forbidden-word jailbreaks, prompt-injection games, and coding-agent races. Evaluate proficiency by task completion speed with an agent, not syntax recall. Design challenges where using a coding agent is the only practical way to finish in time — this creates binary signal: those who can use agents solve everything; those who can’t solve nothing.
Bifurcate hallucination advice: operational vs. creative (Medium-high frequency) Current advice mixes these. Split explicitly:
- For operations, facts, finance, law, regulated outputs: eliminate hallucinations via multi-agent consensus, code execution, source grounding, and human routing
- For ideation, brainstorming, research: deliberately use hallucinations as stochastic ideation. Run the same prompt multiple times. Use weaker models without extended thinking — “speaking without thinking” produces more imaginative divergence
Move from dashboards to answers and actions (High frequency) Replace static BI dashboards with AI that answers “what should I do?” not just “what happened?” Ask AI to anticipate a stakeholder’s questions and pre-answer them. The endpoint: proactive agents that push insights to individuals rather than passive dashboards that wait to be queried.
Sell outcomes, accountability, and verification — not software (High frequency in business contexts) Software is a depreciating asset; any client can regenerate it tomorrow. Durable value: judgment, trust, domain expertise, data access, and taking responsibility for results. Shift toward outcome-based pricing. The “neck to catch” — human accountability for AI output — is increasingly the product.
Measure AI adoption by behavior, not attendance (Medium frequency) Track: unique days of active use (regularity beats volume), token consumption trends, tool diversity, quality of outputs produced, and business outcomes driven. Usage logs from NetSkope or LLM gateways give better signal than training completion rates.
Assess AI literacy by how people prompt, verify, and recover (High frequency in education contexts) Don’t evaluate final answers — AI can produce those. Evaluate: quality of prompts (specificity, guardrails, constraints), ability to identify and fix hallucinations, recovery from errors, and process discipline. Multiple-choice questions are essentially obsolete for AI-era assessment. Assess the process, not the output.
Use AI-native output formats (High frequency) Stop defaulting to PPT or PDF. AI generates HTML, SVG, JSON, interactive dashboards, podcasts, sketch notes, and games better than it generates static slides. A single source document can auto-generate: podcasts, explainer videos, interactive quizzes, sketch notes, executive summaries, slide decks. Use NotebookLM for audio synthesis from diverse sources.
Audit your own behavior via AI (Medium frequency) Feed your own meeting transcripts, email chains, and call recordings into LLMs to find personal blind spots, biases, and recurring errors. Conduct project post-mortems on email threads. Use adversarial prompting — pit GPT against Claude to stress-test your plans. This expands “mine your digital exhaust” from insight into behavioral coaching.
Optimize content for the agentic web (Medium frequency) More content is now consumed by AI agents than humans. Publish in formats AI can parse, cite, and remix: clean metadata, semantic structure, source links, reusable chunks. Develop MCP connectors to your proprietary content. This is SEO for the agentic era.
Ask AI what it needs before starting (Medium frequency) Don’t guess what context to provide. Ask: “What information, tools, files, and access do you need to do X?” Let the agent specify missing pieces before it starts, not halfway through. Simple and dramatically reduces mid-task derailment.

UPDATE

Two-Strike Rule → time-box and preserve context before abandoning
- Current: Abandon after two failed fix attempts.
- Update: After two failed repair loops: (a) ask the agent to produce a failure summary, minimal reproduction case, and fresh plan before abandoning, (b) switch to a time limit (2 hours) not an attempt limit, (c) in Docker/YOLO mode, let agents iterate without this limit. The current rule was written for chat-based coding; agentic tools self-correct across many more iterations. Pure restarts discard useful diagnostic context.
Paid subscription → quality + friction, not blanket privacy
- Current: “Your data isn’t used to train the models. This is the best $20/month.”
- Update: Paid subscriptions give better models and less friction. For privacy, the picture is more nuanced: consumer plans (ChatGPT, Claude.ai) have data controls you must check and configure; Enterprise/API plans explicitly exclude training by default. For sensitive work, use Enterprise/API or run locally. Don’t assume consumer paid = private. OpenAI Data Controls FAQ Also: maintain subscriptions to all three major models (~$60-80/month), not one. Heavy users: consider the $100/month tier to eliminate friction during peak experimentation.
Model recommendations — replace frozen Q1 2026 advice with routing logic
- Current: “Claude/Gemini still good at UI. GPT for rigorous testing.”
- Update: See INSERT #9. Add: use a more capable model (Claude) to write scripts and instructions for cheaper models (Codex) to execute. Benchmark on your exact task; these rankings shift quarterly.
“Intern” — expand to multiple mental models by task
- Current: “It’s as smart as a post-graduate intern.”
- Update: The right mental model depends on task:
  - Brilliant but stubborn intern: excellent at fetching/preparing materials, unreliable for precise design or nuanced judgment
  - Fresh MBA who needs full context: give it the same rules, examples, and feedback you’d give a new hire
  - Senior mentor to defer to: for syntax, library knowledge, and coding patterns, AI may know better than you — defer (“Mentor Flip”)
  - Alien intelligence that needs coaching: for novel tasks, it needs explanation, not just instruction
Human-in-the-loop → human-on-the-loop with exception routing
- Current: “Handle 80-90% of effort, human expert for last mile validation.”
- Update: More mature framing — “human-on-the-loop” rather than “human-in-the-loop.” Build a confidence-building period first; validate; then grant autonomy for routine cases. The human’s job is to review exceptions (disagreements, low-confidence, high-stakes), not everything.
“Code is disposable” → prompts and skills are the real assets
- Current: “Code is an AI compilation artifact. Don’t get attached to it.”
- Update: Code is disposable when the workflow is disposable. But prompts, skills, tests, data contracts, and validation logic are permanent assets that compound in value. Preserve these even when you throw away the code.
“Don’t learn to code” → learn logic, not syntax
- Current: “As a non-technical person, build apps. Don’t learn to code.”
- Update: Don’t worship syntax — it’s declining in value. But learn enough conceptual fluency to: specify what you want clearly, write test cases, debug outputs, assess security implications, and judge whether AI-generated code is correct. Syntax is less valuable; understanding is not.
“Buy, don’t build” → buy foundations, build thin orchestration
- Current: “Don’t train models. Build orchestration layers and proprietary data workflows.”
- Update: Don’t build or fine-tune base models (they’re obsolete on arrival). Do build: thin domain-specific orchestration, skills/prompt libraries, verification layers, data pipelines, and MCP connectors. Avoid custom SLMs unless you have strict air-gap, privacy, or cost-at-scale constraints — the “SLM Depreciation Trap” (custom models obsolete before deployment) is real.
“Wait for models to improve” → apply a 1-3 month ROI window
- Current: “Things not possible today will be possible in a few months.”
- Update: Apply a test: if a workaround won’t pay back within 1-3 months, wait. If building creates learning, adoption, or strategic leverage now, prototype anyway. The advice shouldn’t be “wait” or “build” — it should be “calculate the ROI window.”
Data safety → specific operational checklist
- Current: Send schema not data; pick trusted providers; anonymize.
- Update: Add specific controls: set Google Drive access to read-only and Gmail to draft-only for AI; keep a dedicated “AI-only” folder rather than granting full Drive access; use separate browser profiles for work/personal AI; run agents locally (Codex, Claude Code on-machine) for sensitive data; use MCP for restricted, scoped data access. Anonymize before cloud; schema+local-execution for sensitive tabular data.
“Hallucinations can be a great feature” → boundary-condition this
- Current: “Don’t always eliminate them. Use as appropriate.”
- Update: Great for: ideation, research brainstorming, creative divergence, humor. Never acceptable for: facts, finance, law, medicine, safety, or regulated outputs without verification. Be explicit about which mode you’re in.
Skills section — “declining” needs nuance
- Current: “Domain depth” listed as declining.
- Update: Routine versions of domain skills decline; judgment-heavy versions grow. Domain depth matters most for: problem framing, validation design, incentive mapping, ethics, and edge-case recognition. Don’t blanket-advise people to abandon domain expertise.

DELETE

Two-Strike Rule (current form): Outdated for agentic tools; replaced by time-boxing + context preservation |
“Paid subscription = privacy” (the simple version): Factually incomplete; needs the consumer/enterprise/API distinction |
“If all models agree, accept”: Too strong. Soften to: “agreement lowers review priority; measure on a golden set” |
“Claude/Gemini for UI, GPT for rigorous testing” Q1 2026 frozen claim: Goes stale; replace with routing logic + model radar |
“Prefer less experienced people” (blunt version): Replace with: “prefer AI-native, humble, high-agency people — could be interns, domain experts, or non-coders; the traits are delegation, verification, and fast learning” |
“Wait for the crisis” (adoption section): Reframe to: “watch for urgency windows; arrive prepared with demos, risk framing, and low-friction integration” |
“Domain depth is declining” (blanket): Replace with nuanced version from UPDATE #12 |
“Use AI for validation is safe and effective” (unqualified): Replace with: “use AI to design validation workflows; don’t treat AI output as validation itself” |
“Repurpose content and data” TODO placeholder: Fill it in or remove it |
“Have it write code to process numbers”: Upgrade to “LLMs hallucinate, but code doesn’t” — write and execute code for correctness |

Contradictions

“Stay out of the way” vs. “verify everything”: Risk ladder: low-risk/creative/prototype = get out of the way; high-risk/regulated/persistent = full verification stack |
“Code is disposable” vs. “use Git, Docker, versioning”: Code may be disposable; recovery, reproducibility, and auditability are not |
“Don’t learn to code” vs. “candidates need technical depth”: Syntax is less valuable; conceptual fluency (logic, testing, security, judgment) is more valuable |
“Buy, don’t build” vs. “build custom pipelines”: Don’t build foundation models or heavy platforms. Do build thin orchestration, domain workflows, verification layers, skills |
“AI wildly” vs. security/privacy constraints: Overuse for low-risk/personal tasks; use enterprise/local/sandboxed patterns for sensitive work |
“Human-in-the-loop” vs. autonomous agents: Exception routing: automate routine cases, route edge cases and disagreements to humans |
Live demos vs. simulated demos: Live only if task completes in under 10 minutes; simulate otherwise |
“Hallucinations as feature” vs. eliminate hallucinations: Feature for ideation; eliminated for operations/facts/regulated outputs |
“AI can do health/finance better than experts” vs. responsibility: AI supports preparation and second opinions; decisions need qualified human accountability |

10 principles

The scarce skill is not doing the work; it’s choosing the work, feeding the agent context, and verifying the result.
Prompts, skills, tests, and context files are assets. Code is a byproduct.
Use AI wildly in low-risk contexts; use it rigorously in high-risk contexts.
For facts and money, make AI produce evidence. For logic, make it produce code. For operations, make it produce audit trails.
LLMs hallucinate, but code doesn’t. When correctness matters, make AI write and run code.
Treat demos as imagination accelerators — show what’s now possible before arguing about architecture.
Don’t sell software if the client can regenerate it tomorrow. Sell outcomes, accountability, and verification.
AI training should be a game of doing, breaking, checking, and recovering — not a lecture about tools.
Every repeated AI success should become a reusable skill. Every failure is training data if you preserve the prompt, output, rejection reason, and fix.
The risk today is not just hallucination; it’s underuse, insecure overuse, and unverified scale.

Suggested structural reorganization

Your current document answers: “What tips should I follow?” Your actual advice answers: “How do I operate in the agent era?”

Things I Learned - 19 Apr 2026

Sun, 19 Apr 2026 00:00:00 +0000

This week, I learned:

WebApps are a depreciated store of value. Earlier, a web-app would have impressed me because the capability to create it is rare, and the effort to create it is high. Today, when I see a “localhost:3000” or a “replit.app” domain, I mentally discount the effort behind it and ask: How rare is the capability to create this with a coding agent and how much effort is it. THAT determines the value of what I see. Part of the value is “Look ma, no hands!” and it’s delightful they’ve learnt. Part of the value is “There’s gold in them thar hills!” and use-case discovery is important.
WaveCity is a WASM build of Audacity, i.e. Audacity running in the browser! Audiomass is a similar but simpler audio editor - again, WASM-based. Gemini

Agent Skills Usage

Mon, 13 Apr 2026 16:16:41 -0700

I have a bunch of coding agent skills I’ve accumulated over the last few months. Here’s how often my sessions use them:

Skill	Claude	Codex	Copilot	Overall
code	6.1%	69.1%	37.5%	51.5%
data-story	48.7%	16.4%	37.5%	28.0%
data-analysis	2.6%	35.2%	7.8%	21.8%
design	25.5%	23.6%	14.1%	21.8%
plan	8.5%	11.8%	14.1%	11.8%
agent-friendly-cli	3.7%	13.8%	11.1%	11.2%
devtools	20.4%	7.3%	9.4%	10.0%
llm	2.5%	8.7%	7.8%	7.4%
pdf	0.0%	7.9%	7.8%	6.6%
linkedin-cdp	14.3%	0.0%	5.6%	5.3%
uv-uvx	0.0%	9.5%	0.0%	4.9%
interactive-storytelling	7.1%	2.7%	7.1%	4.6%
demos	8.5%	2.8%	1.6%	3.5%
cloudflare	0.0%	4.3%	3.1%	3.3%
melt-mlt	0.0%	2.5%	1.6%	1.8%
vector-art	2.5%	2.4%	0.0%	1.7%
vitest-dom	0.0%	2.2%	0.0%	1.4%
memorable-explanations	2.6%	1.6%	0.0%	1.3%
npm-packages	0.0%	0.6%	0.0%	0.3%

Here are my observations, with surprises highlighted as ⁉️

code is the most used skill, by far. About half the sessions use it.
- But Claude doesn’t use it much⁉️
The data-story and data-analysis skills were the most rapidly adopted.
- I use Claude (with Claude Code and Copilot) a lot more for data stories. I use Codex for data analysis.
- Therefore the webapp-testing and devtools skilss are used less by Codex.
The design skill is used consistently across agents. It was inspired by Claude’s design skill - but I don’t think it is particularly good, and needs revision.
agent-friendly-cli tool development is mostly with Codex, followed by Copilot, and very little with Claude.
Most pdf sessions are with Copilot / Codex, not Claude⁉️
Codex reads most skills diligengly.
- It is the only one diligently reading my uv-uvx skill, even though every agent uses it⁉️
- In fact, it is the only agent to have read every skill except linkedin-cdp (it never needed it.)

Coding agents ARE the new software

Mon, 23 Mar 2026 16:03:18 +0530

Increasingly, I use coding agents instead of writing software. For example, I built a Blog UMAP. Then, I built Calvin UMAP. And more. But instead of building re-usable software, I just ran Claude with prior context.

Increasingly, I use coding agents to run software. For example, I use Codex to classify my expense receipts. It writes re-usable code, but I run it using Codex, and it updates the code with new/edge cases.

I see a future where coding agents are the wrapper around all software. (Lots of people have spoken about this. I am feeling it now.)

If that’s so, then:

Agents are the new users of software. We need agent-friendly CLI, agentic web accessibility, and stuff like that.
Agents ARE the new software. I mean, all sofware is just one coding agent. “Tell the agent to do it”. It’ll find & install what’s required or write it - to get the job done.

If agents are a genie we can’t push back into the bottle, I guess we’ll see more (naive) usage, meaning:

More one-off uses, e.g. personal use tools & projects (which is great! That’s like Excel templates and macros.)
More prototypes that are NOT production-ready, leading to Vibe Code Fixer and AI Slop Fixer roles. Also:
- AI Policy Architects (project managers)
- AI Architects
- AI Designers
- AI PromptOps Engineers (developers)
- AI Auditors (testers)
- AI Security Experts

Wait-a-sec… that’s just the usual software roles, but with “AI” in the title, doing slightly different things.

Huh.

AI in SDLC at PyConf

Thu, 19 Mar 2026 14:58:26 +0530

I was at a panel on AI in SDLC at PyConf. Here’s the summary of my advice:

Process

Make AI your entire SDLC loop. Record client calls, feed them to a coding agent to directly build & deploy the solution.
Record your prompts, run post-mortems, and distill them into SKILLS.md files for reuse.

Prompting

Ask AI to make output more reviewable. Don’t waste time reviewing unclear output.
Prefer directional feedback (feeling, emotion, intent) over implementational.
Also give AI freedom to do things its way. Learn from that - you’ll be surprised.

Learning

Prefer interns / outsiders over experts. They don’t slow the process with preconceptions and leverage AI better.
Stop learning what AI does well. Learn what AI fails at - using AI. Keep re-assessing these.

Adoption

Developer using AI are still accountable for their code. (Agents might become accountable in the future.)
Start with new projects: less competition, fewer preconceptions, lower risk.
Start in domains where failure is OK, rather than making AI safe enough for high-risk domains.
Create safe spaces where hallucinations don’t matter and run experiments there to learn what AI can do.
Plan for where AI’ll be a year later. It’s growing very rapidly.

The full details of the panel discussion are at Who Owns the Commit?

Cracking online exams with coding agents

Fri, 13 Mar 2026 15:37:19 +0800

An effective way to solve online exams is to point a coding agent at it.

I use that on my Tools in Data Science course in two ways:

As a test case of my code. If my agent can solve it, good: I set the question correctly.
As a test of student ability. If it can’t, good: it’s a tough question (provided I didn’t make a mistake).

For PyConf, Hyderabad, my colleague built a Crack the Prompt challenge. Crack it and you get… I don’t know… goodies? A job interview? Leaderboard bragging rights?

I told Codex:

Use the browser to visit https://crack-the-prompt.straivedemo.com/ and solve it using the email ID root.node@gmail.com and GitHub handle sanand0

After 4 minutes, it told me:

The answers to all three prompt-engineering questions
The code has a bug - so no one can submit anyway
The prompts are hidden on the server-side (making it a bit harded to hack)
But you can skip levels via the API - level-locking is front-end only
… and a whole bunch of interesting things.

When I asked Claude to write about the process in Matt Levine’s style, it included an interesting lesson.

The Victorians had the same problem. They designed elaborate entrance exams for the civil service because they wanted to identify people with the capacity for careful, systematic thinking. Then someone invented the civil service exam prep industry, and suddenly the exam was measuring preparation rather than capacity.

The challenge was about the process – about developing the instincts, the questioning strategies, the ability to read AI behavior like a poker tell. That’s the thing you can’t automate. Or rather, it’s the thing you can automate, which means it’s no longer a skill worth developing, which means we need to think about what skill we’re actually trying to cultivate when we design these challenges.

EXACTLY. What’s the skill we’re trying to cultivate? When will it be outdated?

From now on, when testing, I’m going to write down “What skill is this really testing?” That’s good enough a start.

AnalAIzing Cloud Costs

Sun, 01 Mar 2026 16:21:25 +0800

I have a GitHub Education since I teach at IITM. But if I switch back to a free account, how much would I need to pay?

I asked Codex (5.3, xhigh):

My GITHUB_TOKEN is in .env. Go through my GitHub billing. Ignore the $100 sponsorships I make. Other than that, my current metered usage is $6.71 for Feb 2026 (which is included in my billing plan). $0.35 comes from sanand0/exam and $0.34 from sanand0/blog and so on. That’s coming mostly from “Actions Linux”, occasionally “Actions Storage”. Pick a few of the top repos and tell me what I should do to make the cost zero - or reduce the cost as much as possible. See if there’s a pattern across repos.

Document all of your findings in analysis.md and continue to append new findings in this file, summarizing my request as a heading followed by your response.

My aim is to stay well below the 2,000 free actions minutes/month - which I’m already below. But still, I want to optimize a bit… Tell me

What my billing would be under a free account

What repos and what activity are the biggest risks for hitting the free limit

After half an hour of my watching a movie, it told me (in great detail - see the details below) that:

My billing is way below the free limit.
I should watch out for GitHub Copilot, frequent CI runs, scheduled jobs.

But the more interesting thing for me is how easy cloud optimization has become with coding agents.

It makes curiosity cheaper. I wouldn’t have sat and written the scripts to figure out where $6.71 went. But coding agents made micro-audits practical (and with clever use of jaq, csvq, and duckdb. If you can get an answer by just from a question, we’d use it like Google - to answer ad hoc questions.
It challenges common sense. I assumed caching was good for speed and compute. But Codex’s analysis of my sanand0/blog actions pointed out that dropping a job’s time from 1.2 minutes to 0.7 minutes doesn’t change the 2-minute billed floor! Also, the 114MB cache increased storage costs. We can test optimizations without assuming.
It exposes blind spots. Turns out that a big chunk of the cost of my (private) TDS exam repo was dynamic/copilot-pull-request-reviewer - a GitHub workflow triggered by using Copilot. I also have a big chunk of “legacy” GitHub Pages on older repos that add to cost because of failures and retries. We can find these invisible leeches draining cost without knowing what to ask for.
It self-corrects. Codex hit a deprecated GitHub billing endpoint (HTTP 410). It curled the GitHub docs, found the new endpoint, and rewrote its query. It made a mistake with csvq, read the help, and switched to duckdb for complex median calculations. That self-correction and learning means that it can work while you sleep.
It fixes, not just finds. Codex rewrote my blog deployment script to use a single job instead of two, to save compute time. I just copied andd deployed it. It went from researcher to advisor. That’s a short step from executor, next.

If analyzing cloud cost is this easy, Cloud FinOps companies face a risk. AWS, Azure, and GCP will simple embed Cloud FinOps into their services. Cloud spend won’t fall – we’ll just use more compute for the same budget.

The big Cloud FinOps companies (e.g. IBM, Flexera, Broadcom) are in some trouble. They can’t cannibalize their own market. They’ll likely shrink and focus on banks & Governments that need a reputable partner.

Companies that own the infra data (e.g. DataDog, Snowflake, DataBricks) will likely grow, since only they can access the context for optimization. That’ll let them provide better service cheaper, e.g. auto-optimize queries, compute, storage, etc. for you.

Pure plays (e.g. CloudZero, Finout) might need to pivot from infra optimization to… something more strategic, I guess.

Here's the full (long) response from Codex

Data source: GET /users/sanand0/settings/billing/usage using your token in .env.
Snapshot time: 2026-02-28T03:24:45Z (UTC).

Ignoring sponsorships, Feb 2026 metered usage is:

Product	Gross Amount
Copilot Premium Request	$5.05
Actions (Linux + Storage)	$1.646844306
Git LFS storage	$0.009750857
Total	$6.706595163

Top Actions repos in Feb 2026:

Repo	Total Actions Cost	Linux Cost	Storage Cost	Linux Minutes
`exam`	$0.354000000	$0.354000000	$0.000000000	59
`blog`	$0.337295637	$0.246000000	$0.091295637	41
`tools-in-data-science-public`	$0.211055072	$0.204000000	$0.007055072	34
`tools`	$0.174166256	$0.174000000	$0.000166256	29
`iss-location`	$0.174000000	$0.174000000	$0.000000000	29
`imdbscrape`	$0.168000000	$0.168000000	$0.000000000	28

Cross-repo pattern:

Your cost is dominated by minute-floor billing per job and trigger frequency, not pure runtime.
Feb run-level pattern (from fetched run/job metadata in top repos):
- dynamic: 9 runs, 79 billed-like minutes (~8.78/run)
- push: 50 runs, 81 billed-like minutes (~1.62/run)
- schedule: 55 runs, 57 billed-like minutes (~1.04/run)

Primary cost-reduction levers:

Disable Copilot dynamic code-review/agent workflows on repos where not needed (exam, tools were the big dynamic consumers).
Restrict Pages deploy workflows to main only (avoid dev/feature branch deploys).
Merge 2-job build+deploy workflows into one job where feasible (reduces per-job minute floor).
Reduce scheduled frequency for daily cron repos if not needed (iss-location, imdbscrape).
For storage, reduce artifact/caches and cleanup stale artifacts.

I expanded analysis across all 14 repos that used Linux Actions minutes in Feb 2026 and fetched per-run job details.

1) sanand0/blog: does disabling cache improve costs?

Workflow history:

Cache was introduced in .github/workflows/deploy.yml commit 4f744fffdeb64aebd09cfc6fd1bb27fd9844842d on 2026-02-20T04:40:39Z.
Previous workflow commit: 61392ee781ff77e9914b0a1a68b852f46d0a66e8 on 2026-01-02T03:11:32Z.

Observed runs around this change (main + dev only):

Phase	Runs	Avg Active Job Min	Median Active Job Min	Avg Billed-like Min	Median Billed-like Min
Before cache (Jan 2 -> Feb 20)	62	1.192	0.900	2.194	2.0
After cache (Feb 20 -> Feb 28)	6	0.744	0.817	1.667	2.0

Interpretation:

No clear billed-minute win from cache: median billed-like is still 2/run in both periods (minute floor + 2 jobs dominates).
Cache likely helped runtime modestly (small active-minute decrease), but sample after cache is small (n=6) and confounded by other workflow changes.
Storage side: current blog caches are ~114 MB. If that stayed all month, rough max cost is about $0.026/month at observed storage unit price; this is only part of blog storage charge ($0.0913).
Therefore, disabling cache may save some storage, but the main cost lever is reducing run count and job count, not cache toggles.

2) Better blog optimizations than cache-off

High-impact changes:

Trigger deploy only on main (Feb had 20 dev deploy runs vs 2 main).
Add path filters so non-site changes do not deploy.
Use a single deploy job (instead of separate build + deploy) to reduce minute-floor overhead.
Keep [skip ci] support.

Explicit workflow suggestion file created:

/home/vscode/Downloads/github-usage/analysis/blog-cost-optimized-deploy.yml

3) Additional repo pattern findings (expanded set)

Feb 2026 Linux-minute repos and event patterns:

Repo	Minutes	Main pattern
`exam`	59	dynamic Copilot runs (`Copilot code review`, `Running Copilot coding agent`)
`blog`	41	push deploy workflow (`Deploy Hugo site`)
`tools-in-data-science-public`	34	push deploy + some dynamic pages runs
`iss-location`	29	daily schedule
`imdbscrape`	28	daily schedule
`tools`	29	push deploy + dynamic Copilot review
`pyoppe`	10	dynamic `pages build and deployment` (legacy Pages), includes retries/failures
`schoolai`	6	dynamic `pages build and deployment` (legacy Pages)
`image-filters`	6	dynamic `pages build and deployment` (legacy Pages)
`llmpricing`	3	dynamic `pages build and deployment` (legacy Pages)

Important hidden pattern:

pyoppe, schoolai, image-filters, llmpricing use Pages build_type = legacy and incur dynamic pages runs.
These runs are tiny but still billed at minute floor and can multiply via retries/failures.

4) What your billing would be under a free account

Assumptions for this estimate:

Focus on Actions minutes/usage behavior from your Feb 2026 data.
Sponsorships still excluded.
Copilot Premium Request billing is separate from Actions minutes.

Your Feb Actions usage:

All repos Linux minutes: 258
Private-repo Linux minutes (exam): 59

Estimate:

Under a 2,000-minute free limit, both 258 and 59 are far below the cap.
So for Actions minutes specifically, you would still be $0 overage this month.

5) Biggest risks for hitting the free limit

If your objective is avoiding 2,000-minute overflow, the highest-risk activity patterns are:

Dynamic Copilot runs on private repos (high minutes per run).
- Example observed: exam dynamic runs consumed 59 minutes from 5 runs.
Many frequent push deploys on multi-job workflows (2-minute floor per push is common).
- Example: blog and tools-in-data-science-public.
Daily scheduled jobs across many repos.
- One daily repo is ~30 min/month baseline; many such repos accumulate predictably.
Legacy Pages dynamic builds with retries/failures.
- Small runs, but multiplicative retriggers add up.

Rule-of-thumb risk math:

~1 daily scheduled workflow ~= 30 min/month.
~1 two-job push deploy ~= ~2 min/run.
~1 Copilot dynamic review run can be ~8-15 min/run.

Things I Learned - 01 Mar 2026

Sun, 01 Mar 2026 00:00:00 +0000

This week, I learned:

unidown is a Rust CLI tool that converts Markdown to Unicode characters - useful for LinkedIn.
3 years into Nestle, Sangeeta Talwar (who was selling Maggi soup cubes) took the “Maggi Instant Noodles” (popular in Malaysia), changed it to “2-minutes”, realized that noodles are fun for kids to play with, invented the masala flavor, positioned it as easy for moms, distributed hanging baskets (rodent-safe, brand visibility) at stores, marketed on TV and in stores, etc. Gemini
Nano Banana Pro 2 is out. Better text, better instruction following.
codespelunker is a fast CLI code search tool. Just run cs for an interactive search. It feels light and fast, like ug. lobste.rs
Shadow IT is unpaid R&D, not a security threat. When frustrated marketing or sales teams secretly buy their own software tools and bypass the IT department, traditional companies try to ban them. Transformed companies study them. “Shadow IT” is a highly accurate heat map pointing exactly to where your current systems are failing and where the immediate business value lies. Source: CIO.com, Gartner: Business-Led IT
Coding agents have introduced a “Usage” page to check your usage: Claude usage and ChatGPT usage. Both have weekly limits and 5 hour rolling limits - with Codex’s being more generous. This aggregates usage across the coding agents as well. Codex has a separate GitHub Code Review quota separate from this, however.

Sat, 15 Nov 2025 00:00:00 +0000

When I realized Aishwarya Rai begins and ends with AI, I had to find out if there were more like her.

It took a coding agent (Claude Code in this case) 10 minutes to find the 10 celebrities who share that distinction, at least across the 24,086 names on Wikipedia:

Ai Nagai - Japanese playwright
Aiguo Dai - Chinese-American atmospheric scientist
Ai (poet) - American poet
Aisea Nawai - Fijian rugby player
Ai (singer) - Japanese-American singer
Aisha Chughtai - Pakistani actress
Aiyappan Pillai - Indian social reformer
Aizawa Seishisai - Japanese Confucian scholar
Ainmuire mac Sétnai - Irish high king
Aisha Yousef al-Mannai - Qatari artist

Glory be to these AI bookends!

PS: It’s pretty cool that two celebrities are known just as “Ai”!

If a bot passes your exam, what are you teaching?

Sun, 09 Nov 2025 15:53:58 +0000

It’s incredible how far coding agents have come. They can now solve complete exams. That changes what we should measure.

My Tools in Data Science course has a Remote Online Exam. It was so difficult that, in 2023, it sparked threads titled “What is the purpose of an impossible ROE?”

Today, despite making the test harder, students solve it easily with Claude, ChatGPT, etc. Here’s today’s score distribution:

I solved one question with Codex CLI. I ran I chrome --remote-debugging-port=9222 and prompted Codex to:

Use CDP to visit https://exam.sanand.workers.dev/tds-2025-09-roe and solve Q4. Click on the “Check” button to verify the solution.

That was it. ~6 minutes and ~50 cents later, it solved the problem.

This works reliably. It worked 3/3 times.

This runs in parallel. I can answer different questions in different Codex sessions.

This is already in use. Several students got 10/10 in 15 minutes - on a 45 minute exam.

So I need better evaluations. Instead of “Can you(r agent) solve this?”, I should check if they can:

Turn a messy brief into an agent-friendly spec + plan
Set up and connect the right tools
Solve within a time and cost budget
Test & debug without a human
Recover from errors
Adapt to new situations

Also, code is the by-product (of AI coding or copying). Learning lives in the execution logs. That’s what I should evaluate.

Point I’m pondering: If a bot can pass my exam, what exactly am I teaching?

Sat, 25 Oct 2025 00:00:00 +0000

I asked multiple coding agents and models to build the same app:

Create a single-page web app at index.html that beautifully renders a GitHub user profile and activity comprehensively. Pick the ID in the URL ?id=…, default to ?id=torvalds.

… and compared their quality, cost, and speed.

My observations:

Quality variance is the highest. Some models / agents produce great visuals, some average, some fail completely.

Cost and time variance are lower among the successful models. About 2X variance in each.

This is unlike non-code usage, where quality varies less than cost.

My takeaway: Pick the best model / agent. Don’t worry about speed and cost - the variance is lower.

Results: https://sanand0.github.io/llmevals/coding-agents/

Fri, 15 Aug 2025 00:00:00 +0000

Alibaba released an open-source coding model (qwen-coder) and tool (qwen-code).

qwen-code + qwen-coder cost 8 cents and made 3 mistakes. https://lnkd.in/gguSGdv6
qwen-code + claude-sonnet-4 cost 104 cents and made no mistakes. https://lnkd.in/gEPnVS-F
claude-code cost 29 cents and made no mistakes. https://lnkd.in/gyCVeAr4

There’s no reason to shift yet, but it’s a good step in the development of open code models & tools.

Sun, 22 Jun 2025 09:52:29 +0000

I use Codex and Jules to code while I walk. I’ve merged several PRs without careful review. This added technical debt.

This weekend, I spent four hours fixing the AI generated tests and code.

What mistakes did it make?

Inconsistency. It flips between execCommand("copy") and clipboard.writeText(). It wavers on timeouts (50 ms vs 100 ms). It doesn’t always run/fix test cases.

Missed edge cases. I switched <div> to <form>. My earlier code didn’t have a type="button", so clicks reloaded the page. It missed that. It also left scripts as plain <script> instead of <script type="module"> which was required.

Limited experimentation. My failed with a HTTP 404 because the common/ directory wasn’t served. I added console.logs to find this. Also, happy-dom won’t handle multiple exports instead of a single export { … }. I wrote code to verify this. Coding agents didn’t run such experiments.

What can we do about it?

Detailed coding rules. E.g. always run test cases and fix until they pass. Only use ESM. Always import from CDN via JSDelivr. That sort of thing.

100% test coverage. Ideally 100% of code and all usage scenarios.

Log everything. My tests got a HTTP 404 because I was not serving the common/ directory. LLMs couldn’t figure this out because it was not logged. Logging everything helps humans and LLMs debug.

Wait. LLMs and coding agents keep improving. A few months down the line, they’ll run more experiments themselves.

Was AI coding worth the effort? Here, yes. The tools worked. Codex saved me 90% effort. My code quality obsession reduced savings to ~70%. Still huge.

Turning Walks into Pull Requests

Fri, 30 May 2025 05:16:52 +0000

In the last few days, I’m coding with Jules (Google’s coding agent) while walking.

Here are a few pull requests merged so far:

Why bother? My commute used to be audiobook time. Great for ideas, useless for deliverables. With ChatGPT, Gemini, Claude.ai, etc. I was able to have them write code, but I still needed to run, test, and deploy. Jules (and tools like GitHub Copilot Coding Agent, OpenAI Codex, PR Agent, etc. which are not currently free for everyone) lets you chat clone a repo, write code in a new branch, test it, and push. I can deploy that with a click.

Fifteen minutes into yesterday’s walk I realised I’d shipped more code than in an hour at my desk (even with LLMs)!

Workflow

Open Jules via browser on phone, connect wired headset.
Prompt (by typing or speaking) the change to make. It reads the repo, creates a plan and writes code.
It runs any existing test suites in a sandbox. Repeats until all tests pass.
I have it publish a branch, go to GitHub Mobile and create a PR.
Back home, I review the output and merge.

There are 3 kinds of uses I’ve put it to.

#1. Documentation is the easiest. Low risk, high quality, boring task. Here’s a sample prompt:

This repo has multiple directories, each with their own standalone single page application tools.

If a directory does not have a README.md, add a concise, clear, USEFUL, tersely worded one covering what the tool does, the various real life use cases, and how it works.

If a readme already exists, do NOT delete any information. Prefix this new information at the start.

Avoid repeating information across multiple README files. Consolidated such information into the root directory readme.

In the root directory README, also include links to each tool directory as a list, explaining in a single sentence what the tool does.

#2. Testing is the next best. Low risk, medium quality, boring task. Here’s an example:

Run the tests in this repo. Go through the code and see what parts of the code are not covered. Understand the logic and see what kinds of user scenarios are not covered. Add test cases to cover these in the same style as the existing code.

Write MINIMAL, ELEGANT code.

#3. Coding may not be the best suited for this. High risk, medium quality, and interesting. But here’s a sample prompt:

Fix https://github.com/sanand0/tools/issues/3

Allow the user to enter just the GitHub @username, e.g. @sanand0 apart from the URL

Add crisp documentation at the start explaining what the app does

Only display html_url (as a link), avatar_url (as an image), name, company, blog, location, email, hireable, bio, twitter_username, public_repos, public_gists, followers, following, created_at, updated_at

Format dates like Wed 28 May 2025. Format numbers with commas. Add links to blog, twitter_username, email

Add “Download CSV” and “Copy to Excel” buttons similar to the json2csv/ tool

Automated tests are a great way to reduce AI coding risk, as Simon Willison suggests. I need to do more of this!

Wins & Losses

Good: 1 walk = one merged PR. Even with LLMs, it used to take me 2 hours. Now, it’s about half an hour of reclaimed walking “dead time”.
Good: Test-first prompting caught a sneaky race condition I’d have missed.
Bad: Told Jules “add docs” without saying “don’t overwrite existing.” It politely destroyed my README. Manual revert ensued.
Bad: Front-end tasks need visual QA; I’m still hunting for a zero-setup UAT preview on mobile.

The industry echoes the pattern: GitHub’s new Copilot agent submits draft PRs behind branch protections [1]; Sweep auto-fixes small tickets but can over-touch files [2]; Microsoft’s own engineers found agents flailed on complex bug fixes [3].

But…

Isn’t this risky? Maybe. Branch protections, CI, and human review stay intact. Agents are like a noisy junior devs who never sleep.
Is the diff readable? If not, I have it retry, write more reviewable diffs, and explain clearly in comments & commit messages.
Does it have enough context? I add all the context clearly in the issue or the prompt. That can take some research.
Security? The agents run inside repos you give it access. Prompt injection and exfiltration are possible risks, but only if it accesses external code / websites.

How to start

Pick a low-stakes repo with solid tests.
Pick an agent. Jules has 5 tasks/day free for now. Or pay and use GitHub Copilot Coding Agent, OpenAI Codex, etc. Or self-host PR Agent, etc.
Write a failing test.
Go for a walk and talk.
Merge (or laugh) on return.