exam-strategy on S Anand

Let AI take your exams

Fri, 12 Jun 2026 08:10:56 +0530

At 2 pm IST today (Fri 12 Jun 2026), I conducted a workshop at Paradox, IITM - at DOMS 101.

My core message is: “AI can solve exams and help you learn. Delegate what AI can do. Learn what AI can’t do instead.”

My talks page for “Let AI take your exams” includes:

The full story + transcript + audio
How Codex solved a real exam, live
My collection of AI-learning techniques - which was not covered in the workshop, but is a useful reference

Here are the takeaways from the workshop:

AI is more capable than you think — and getting smarter. Recalibrate constantly what it can and can’t do. Note down what it can’t, because that is precisely where your value lives.
Delegate first; learn the rest. Give everything to AI. Focus your learning on what it can’t yet do — that’s where the value will be. It’s a moving filter; revisit it every quarter.
Always use the best model, turned up high. Reserve “fast and cheap” for the ~5% of moments you need a quick answer. And remember students get serious AI free via the GitHub Student Pack and Gemini.
Make the AI ask you for context. “If you need more information, ask me.” You don’t have to know what context it needs — push that burden back to the machine.
Beat hallucination with a maker-checker. Two independent models that must agree cut errors from 14% to under 4%. Tell the checker to “find the errors,” not to grade.
Loop with feedback in verifiable environments. Point an agent at an exam, a codebase, anything that scores itself — let it try, submit, read the result, retry. This is the most powerful technique AI has.
Calibrate, don’t just trust. Practise predicting whether AI will get something right — even on topics you don’t know. Watch for base-rate traps and familiar problems with one changed premise.
Be lazy, productively. Don’t read AI’s 20-page output — train it to give you five words. Working well with AI is a management skill.
Learn from peers. Multiple people trying things is how you discover what works. Non-transactional relationships are the rising currency of the AI era.
Apply the scientific method to everything. Form a hypothesis, hunt for evidence, try to falsify yourself. And when a system blocks you unfairly — hack it, then publish what you learned.

Here are the takeaways from how Codex solved the exam:

An agent operates the environment; a chatbot answers the question. Codex read source, ran code, clicked Check, and looped on feedback. That’s why it beat copy-paste — the exam was full of affordances a chatbot can’t touch.
Verifiable environments favour AI. The more checkable the exam — validators, error strings, downloadable files, a live Check button — the more it helped the agent, not the student. “AI-proof” and “feedback-rich” are opposites.
Most failures were wording, not reasoning — and the source fixed them. The fix for a brittle validator was to read the validator. Pass the error message back to the agent and it converges.
Don’t guess where attempts are limited. The network game punished early guesses. The recoverable mistakes had feedback; the costly ones didn’t. Triage cheap-and-certain first.
The gap between 9 and 10 was a credential, not a brain. Same model, same skill. Anand’s missing mark was an invalid token. In the AI era, “can it?” often means “does it have the keys?”
It cost about a coffee. ~$2–3 of tokens for the whole exam, ~96% cached. The real cost is the human judgment to know when the agent is plausibly, confidently wrong.

Original announcement

You can join online at https://meet.google.com/cpt-faee-ucx and ask questions on chat.

Agenda:

You’ve been told AI can pass your exams. But what happens when you actually watch it try — live, on your questions, in real time?

This workshop starts with a collective experiment: we ask coding agents to solve real exams (including IITM exams) and see how it solves them.

What follows isn’t a tutorial on prompting — it’s an autopsy that reveals what your exams are actually testing, where AI confidently hallucinates, and what that means for what’s worth learning.

You’ll leave with a reframed understanding of your degree (the goal isn’t answers, it’s the ability to catch wrong ones) and a concrete study rituals that uses AI as a Socratic sparring partner rather than an answer machine.

Come with a question you got wrong recently — it’s going to be useful.

Real agenda: An ask-me-anything session plus real-life experiments.

How to create a data-driven exam strategy

Thu, 09 Oct 2025 01:24:19 +0000

Can ChatGPT give teachers data-driven heuristics on student grades?

I uploaded last term’s scores from about 1,700 students in my Tools in Data Science course and asked ChatGPT:

This sheet contains the scores of students … (and explained the columns).

I want to find out what are the best predictors of the total plus bonus… (and explained how scores are calculated).

I am looking for simple statements with 80%+ correctness along the lines of:

“if ga1_f + ga_4f > x then you will not fail; this is true for 90% of the students” or

“your total will effectively be your p1_f score, within plus or minus 5 marks; this is true for ~90% of the students” or

“if you get below 40% in ga3_f then you will likely fail; 80% of the students who did so fail.”

Earlier predictors (GA1, GA2, maybe GA3) are more useful (to students) …

Two kinds of students: Those who want to avoid failing and those who want to score high….

.. give me the best student advice. Phrase it conversationally and include numerical evidence in simple language.

After a 30 minute conversation, ChatGPT delivered concrete, actionable advice:

One-page student playbook

If you’re behind / “don’t fail”:

Immediate target: GA2 > 45 and GA3 > 55. That alone cuts fail risk to ~16%.
If possible, push >85/85 to get fail to ~8%.
Translation: stop skipping; get partial credit; move into the “safe” GA2/GA3 band.

If you’re aiming for an A:

Path A1 (broad coverage): P2 + ET >= 180 > A ~ 35% (n=169).
Path A2 (also broad): P1 + P2 >= 180 > A ~ 45% (n=161).
If early GAs are fine (GA2/GA3 >=70), keep them steady, but your marginal effort should go to P2 and ET.

If you’re aiming for an S:

Path S1 (most reliable): P2 >= 90 + ROE >= 80 > S ~ 77% (n=26).
Path S2 (elite): P2 >= 90 + ET >= 85 + ROE >= 80 > S ~ 91% (n=11).
Simple rule of thumb: P2 + ROE >= 170 or ET + ROE >= 160 > ~70% S with 0% fails in those groups.
Translation: ace ROE. Use P2 and ET as the second and third pillars.

Students: Don’t study harder. Study what moves the marks. (Learning be damned!)

Teachers: Try this with your course data!

Full chat: https://chatgpt.com/share/68e7034a-7340-800c-b259-0bd51715ad11

Implicit information

Thu, 03 Jan 2008 12:00:00 +0000

From what I’ve seen, puzzles and exam questions share two un-real-worldly characteristics. Firstly, you are guaranteed that a solution exists. Secondly, you are given that all the information provided to you is relevant. (Well, not always. Some case studies I’ve seen have had their share of contrived irrelevance. But that’s often what it is, I think. People fill in the relevant stuff, and then try and distract by adding irrelevant material in the hope of making it more real-world-like. But that’s just a guess).

These are very powerful constraints. I know of nothing that has given me as much confidence in solving puzzles as the assurance that a solution exists (and that someone thinks me capable of getting it).

But it’s more than just a confidence builder. The guarantee that a solution (and invariably it’s a unique) is a very powerful one. An extreme case is an objective type question, which explicitly provides three guarantees:

There is a solution
There is only ONE solution
It is among the choices listed below

(Some papers try and take away the first guarantee by having an (E) None of the above category. But that’s still leaving behind the other two more powerful guarantees.)

Marking answers randomly, or marking (A) for every question would still get you 25% in an exam with 4 choices. (Marking (C) would prove just as good, unless you had a kind professor like this.) That’s better than any real-world scenario I’ve seen. (Real-world strategies aren’t much better, though.)

Using guarantee 2, you can eliminate choices easily. If (A) and (B) do not satisfy some property of the solution, they CANNOT be the answer. There’s only one solution, and these are not it.

Using guarantee 3, you can pick the last remaining choice wihout having to check it. The solution is definitely among the choices listed. So you don’t need to solve an objective type question. You just need to pick the right answer – which is completely different.

The principle applies even outside of objective type questions, especially in mathematically-oriented problems, or puzzles. And you can solve it by trial and error. For example, try this one from Martin Gardner’s Mathematical Magic Show:

Two brothers own n sheep, each of which is sold for n dollars. Thus they have n2 dollars in all. This is in the form of 10 dollar notes and 1 dollar coins, the number of 1 dollar coins being less than 10 dollars. The elder brother divides the money as follows: he takes a note for himself, gives one to his younger brother, takes a note for himself and so on. At the end, the younger brother complains that the elder took the first note as well as the last. So the elder gives the younger all the one dollar coins. The younger brother complains that he still has more. So the elder brother writes the younger a cheque to equalize their share. What was the cheque for?

Now, this is a weird problem. Think about it. You’re told almost NOTHING. And you have to guess what the amount is. (Note: you don’t have to guess what ’n’ is. That’s impossible.)

Here’s how I solved the problem. I said, let me find even one case where the elder brother gets the first and last note. Let’s see what the answer is. Whatever the answer is for that case, it has to be the answer for all other cases – because otherwise, the problem does not have a unique solution.

So I tried n=1. n=2. n=3. For n=4, the amount is 16. That’s 1 $10 note and 6 $1 coins. The elder brother would get the first and the last $10 note. The younger would get $6. So the elder would have $4 more than the younger, and would write out a cheque for $2. (It’s amazing how many people get as far as the $4, but forget to divide by two.)

You can try if for any other value that has an odd number of $10 notes. It has to be for n ending with 4 or 6. That means n2 ends in 6, and the cheque has to be for $2.

Notice that you didn’t need number theory to get the answer. The assurance that there is a unique answer is enough.

There’s another kind of implicit information usually available: the amount of information there is. For example, take the following question:

Which city has a higher population: San Antonio or San Diego?

Children in the US apparantly had difficulty answering it. Children in Germany had less trouble. The reason? The German kids had heard of San Diego, but not San Antonio. They figured the one they’d heard of was more likely bigger. Knowing less may be better.

It’s the same principle you use to check spellings. Run a Google search on two spellings. The one that returns a higher number of results is the correct spelling. (Of course, Google has a spelling correction mechanism that works well, but I use it for Tamil words. I can never tell if I should use ர or ற.)

Of course, the fundamental assumption here is: MORE INFORMATION = MORE CORRECT, which is not always the case. But the point I’m driving to is this:

You’re always given additional information. Even if you’re not given any information, that’s informative.

Comments

Sathyaraj 3 Jan 2008 12:00 pm: this article was “fucking” good..i was reading a book called fooled by randomness and i just could relate to the way you and the author went about writing this.. can you write more about monte carlo simulation too?? and how it is used in finance…
Vasanth 3 Jan 2008 12:00 pm: Very interesting post! New insight for me into puzzles. I had always realized, but never analyzed, that I never give up when it comes to puzzles and brain teasers. On the contrary I give up relatively easily with some technical or work related problem the minute it appears probable that a (reasonable) solution does not exist.
S Anand 3 Jan 2008 12:00 pm: @Sathyaraj: Thanks! The sequel to “Fooled by Randomness” is pretty good to: “The Black Swan”. @Vasanth: Another big problem, if you ask me, is that the problem is unbounded. In puzzles, you have a test case to determine whether you’ve solved it or not. In life, you aren’t often sure.
Jayanth Sankar 3 Jan 2008 12:00 pm: Hey I like your posts. Pretty good. The question about the money between the two brothers.. I have a doubt here.. you had mentioned number of 1 dollar notes should be less than 10 dollar notes, so then 16 wont work right ? 1 - $ 10 and 6 - $1… what am I missing ?