codex-cli on S Anand

If a bot passes your exam, what are you teaching?

Sun, 09 Nov 2025 15:53:58 +0000

It’s incredible how far coding agents have come. They can now solve complete exams. That changes what we should measure.

My Tools in Data Science course has a Remote Online Exam. It was so difficult that, in 2023, it sparked threads titled “What is the purpose of an impossible ROE?”

Today, despite making the test harder, students solve it easily with Claude, ChatGPT, etc. Here’s today’s score distribution:

I solved one question with Codex CLI. I ran I chrome --remote-debugging-port=9222 and prompted Codex to:

Use CDP to visit https://exam.sanand.workers.dev/tds-2025-09-roe and solve Q4. Click on the “Check” button to verify the solution.

That was it. ~6 minutes and ~50 cents later, it solved the problem.

This works reliably. It worked 3/3 times.

This runs in parallel. I can answer different questions in different Codex sessions.

This is already in use. Several students got 10/10 in 15 minutes - on a 45 minute exam.

So I need better evaluations. Instead of “Can you(r agent) solve this?”, I should check if they can:

Turn a messy brief into an agent-friendly spec + plan
Set up and connect the right tools
Solve within a time and cost budget
Test & debug without a human
Recover from errors
Adapt to new situations

Also, code is the by-product (of AI coding or copying). Learning lives in the execution logs. That’s what I should evaluate.

Point I’m pondering: If a bot can pass my exam, what exactly am I teaching?

Things I Learned - 26 Oct 2025

Sun, 26 Oct 2025 00:00:00 +0000

This week, I learned:

Before founding a place to do good, work in a place that does good and learn. Ben Werdmuller
What should we teach when vibe coding becomes good enough for non-coders? Ethan Mollick
- Problem decomposition
- Clear communication & spec writing
- Core technical foundations: file systems, access control, networking, APIs, version control, data structures, databases, deployment
- Software development skills: Debugging, Testing, Refactoring, Design patterns, UI/UX
- Project management: requirements, prioritization, scoping, …
Codex CLI tips:
- codex --add-dir $DIR lets you write into $DIR
- codex --full-auto is the equivalent of codex --sandbox workspace-write --ask-for-approval on-request
Terse code is not necessarily easier or harder for LLMs to write. It’s about how unusual (or not aligned with training data) the code is. Gabi Teoduru
How are people using browser agents like Comet / Atlas? Simon Willison
- Most popular: YouTube video summaries with timestamps
- Most useful: Form filling: Government forms, data entry, repetitive bureaucratic tasks
  - Foreign language navigation: Applying for pension in Korea, navigating sites in other languages
  - Time reporting auto-completion
  - Insurance claims: Reading policy documents and drafting appeals (successfully got claim reimbursed in India)
  - Compliance training click throughs
- Next most useful: Shopping / planning
  - Energy provider comparison - Comet checked current plan vs competitors on Check24, calculated exact annual savings per provider
  - Financial tracking: Finding Amazon orders, tracking Airbnb spending with refund calculations, analyzing bank transactions
  - Trip planning: Mapping 50-100 places on Google Maps automatically
- Interesting: Airport shuttle discovery - Found shuttle that user missed in manual searching
HubFS mounts GitHub repos on the file system. Every file system action directly works on GitHub via a REST API. Useful for some scenarios but less useful for note-taking than something like GitDoc which offers a delayed sync.
Ernest Ryu solved an open problem in convex optimization using ChatGPT. Quotes:
- ChatGPT is now at the level of solving some math research questions, but you do need an expert guiding it.
- ChatGPT was really effective at accelerating my progress. This work took about 12 hours, spread over 3 days. In hindsight, the proof is really simple.
- But I iterated through so many other strategies that didn’t pan out, and ChatGPT crucially helped to quickly explore and eliminate those dead-end approaches. Also, the key successful steps were suggested by ChatGPT.
- ChatGPT did not produce the proof in a single prompt. The process was highly interactive. It generated many arguments, roughly 80% of which were incorrect.
- Yet some were genuinely novel to me. Whenever I recognized a novel idea, whether correct or only partially so, I distilled the key insight and prompted ChatGPT to develop it further.
- My contribution:
  - Filtering out incorrect arguments and accumulating a set of correct facts.
  - Identifying promising new lines of reasoning and guiding ChatGPT to explore them further
  - Recognizing when a strategy had been fully explored and deciding when to move on.
- ChatGPT’s contribution:
  - Producing the final proof argument.
  - Significantly accelerating my (or our) exploration of the many dead-end arguments, rapidly ruling out approaches that did not work.
Comparing the GPT 4.1 and 5 models at all different of reasoning, I’ve switched my default from GPT 4.1 mini to GPT 5 mini (medium). Far smarter for a slightly higher cost. Artificial Analysis
python -m pdb -c continue script.py or uv run -m pdb -c continue script.py runs a script and drops into pdb on unhandled exceptions (post-mortem). ChatGPT
Technology removes constraints. We then do what we really value. Claude
- When writing became digitized, we stopped cared about spelling/handwriting for its own sake. Spelling bees and handwriting classes declined. “ur” is acceptable.
- When fitness tracking became easy, many just track, few exercise more. Few people value exercise
- When GPS became ubiquitous, we stopped learning geography. Most value arriving, not knowing
- When photography became unlimited, most captured moments. Few perfected shots
I had Codex scrape my ~2,000 pending invites on LinkedIn and asked ChatGPT to analyze it. Here are learnings: ChatGPT, private
- Power-law. 5% of inviters account for ~42% of all common connections. Top 10 people alone for ~20%.
- IITM student invites are high (~14%), but with 0-2 common connects, i.e. distant strangers.
- EdTech is tiny in count but has the highest common connections per person (outlier-sensitive but real).
- Among ≥20-commons, many hold VP/Head/Site-Lead titles in Data/AI or GenAI (not just recruiters).
- GenAI people are 7-8% and steady across months. Not a useful signal to prioritize.
- Premium ~ Senior. Premium accounts show ~40% senior titles vs ~29% for non-premium.
- Finance invites have higher seniority rate and more common connects than healthcare.
- Followers have higher common connections (~6 vs ~4).
⭐ Memory can be code. Agent memory is anything it choose to persist. Agents can write code on the fly to automate tasks, save them, and serve the code on the next request, potentially modifying the code as required. This is like the conscious mind saving a habit for the subconscious to execute fast.
Finally: Microsoft Office has an agent mode that lets you talk to it and do stuff. The Verge

Measuring talking time with LLMs

Sun, 24 Aug 2025 06:26:22 +0000

I record my conversations these days, mainly for LLM use. I use them in 3 ways:

Summarize what I learned and the next steps.
Ideate as raw material for my Ideator tool: /blog/llms-as-idea-connection-machines/
Analyze my transcript statistics.

For example, I learned that:

When I’m interviewing, others ramble (speak long per turn), I am brief (less words/turn) and quiet (lower voice share). In one interview, I spoke ~30 words per turn. Others spoke ~120. My share was ~10%.
When I’m advising or demo-ing, I ramble. I spoke ~120 words per turn in an advice call, and took ~75% of the talk-time.
This pattern is independent of meeting length and group size.

I used Codex CLI (command-line tool) for this, with the prompt:

Go through the transcripts in this folder and estimate the % of time Anand was speaking vs others, by conversation.

Then I prompted for correlations and interpretations. This combines three things I find powerful:

LLMs writing & running code
LLMs interpreting the results
Running on local data in my machine

LLMs working on local docs (not data) is new to me. I plan to do much more with it.