assessment-design on S Anand

TDS Jan 2026 ROE

Sun, 05 Apr 2026 19:04:37 +0800

Tools in Data Science has a remote online exam (ROE). It has a tough reputation. We conducted one today.

Here’s how today’s ROE unfolded.

The TAs had created 13 questions and shared it with me yesterday. This morning, I tried solving them.

At first glance, it looked scarily hard! But I just jumpted down a few questions, and found that five questions were trivial, i.e. I just used the “Ask AI” button to copy the question into ChatGPT and it gave me the answer.

🟢 06. Layered Encoding Challenge (2.0) is one-shot: ChatGPT
🟢 07. Region Containing Point (1.0) is one-shot: ChatGPT - see second chat
🟢 10. Fix Broken JSON File (1.0) is one-shot: ChatGPT
🟢 11. Cross-entity disambiguation (1.0) is one-shot: ChatGPT
🟢 13. Record Terminal Session with asciinema (0.5) is one-shot: ChatGPT

For another four, I needed to just make sure I uploaded the files or HTML:

🟡 03. Regex Golf Challenge (2.0) is one-shot but it’s better to download and upload the text files, maybe, than copy-paste: ChatGPT
🟡 04. Maze Solver with Constraints (2.0) is one-shot but needs you to upload the image and a good vision model: ChatGPT
🟡 05. Cipher Trail (2.0) is one-shot but needs you to provide the secret HTML: ChatGPT
🟡 12. Simple Question (0.5) needs you to provide the secret HTML: ChatGPT

Two questions confused ChatGPT a bit, and it needed some nudges. There is real learning here.

🟠 08. Reorganize Files with Shell Commands (1.0) is hard because of Unicode issues and the extra README.md students should delete. Asking for variations is the learning: ChatGPT
🟠 09. Refactor Python Code with VS Code (1.0) takes few attempts (question is imperfect, intentionally) but error messages are excellent, so iterative feedback is the learning: ChatGPT

One was pretty hard and ChatGPT struggled with it.

🔴 02. Korean Speech Dataset API Validation (5.0) actually requires work. At first, ChatGPT refused ethically. When reframed, it tried, but the human-in-the-loop (me) was too slow. So I used Codex, which literally hacks towards the solution! It searched online for existing solutions, read the GitHub discussions for this topic, found my browser tab and started testing itself, … and solved it in 10.5 min!

This leaves the one question that AI can’t solve:

🔴 01. Collaborative Token Exchange (5.0) asks you to collect “tokens” from other people and share it. I asked Codex to hack it, but after an hour (of logging into my personal account, my IITM account, even my father’s account, exhausting my token limits, and totally psyching me), it declared the question unhackable.

Based on this, the instructors, teaching assistants and I decided that:

This exam is too easy with AI help. Combined with collaboration, it’s ultra-easy. Therefore, let’s add some old questions:
- 14. FastAPI Time Series Caching (0.5)
- 15. AI Video Attendee Extraction (0.5)
Thanks to Project 1, people already collaborate at scale. So let’s ask for 500 tokens instead of 100.

We deployed the exam at 12:00 pm IST, an hour before the scheduled time.

The hackers (e.g. who scan the source code, or change their system clocks) could see the questions earliy and started sharing and solving them.

The TAs said, “Anand, shall we add more questions to make it tougher?”

I said, “No, it’s OK. The ROE has built a reputation for difficulty. Let that change. Let them have an easy exam.”

“If they’re going to split this in groups and have coding agents solve it, they’ll score full marks in 10-15 minutes,” I said to myself.

When the exam ended, this was the score distribution.

There were several surprises here. Firstly, 9 questions were repeats. Yet, barring one question, they scored lower, though they appeared in equally tough ROEs in the past.

#	Question	%	Previous %	Previous exam
14	⚪ FastAPI Time Series Caching	8%	39%	2025 Sep ROE
15	⚪ AI Video Attendee Extraction	12%	64%	2026 Jan GA3
11	🟢 Cross-entity disambiguation	13%	64%	2026 Jan GA4
7	🟢 Region Containing Point	15%	38%	2025 Sep ROE
9	🟠 Refactor Python Code with VS Code	17%	57%	2026 Jan GA1
13	🟢 Record Terminal Session with asciinema	21%	66%	2026 Jan GA1
10	🟢 Fix Broken JSON File	30%	64%	2026 Jan GA1
12	🟡 Simple Question	41%	64%	2026 Jan GA1
8	🟠 Reorganize Files with Shell Commands	53%	38%	2026 Jan GA1

Just as surprisingly, they scored higher than these on 3 of the 5 new questions:

59%: 🟡 03. Regex Golf Challenge (2.0) (253 / 430)
59%: 🟡 04. Maze Solver with Constraints (2.0) (252 / 430)
54%: 🟡 05. Cipher Trail (2.0) (233 / 430)

One new question ended up being almost the hardest question - despite it being one-shot-table for ChatGPT (GPT 5.4, extended thinking).

2%: 🟢 06. Layered Encoding Challenge (2.0) (9 / 430)

The toughest, though, had a 1% success rate. Though Codex could solve it in 10 min, it’s a genuinely hard question.

1%: 🔴 02. Korean Speech Dataset API Validation (5.0) (4 / 430)

Which leaves us with the collaboration question - the one that AI can’t solve.

31%: 🔴 01. Collaborative Token Exchange (5.0) (132 / 430)

This question is a whole new dynamic altogether. There were about 5 clear clusters of students, ranging from 5 - 35 students, who were collaborating. They were trading bundles of tokens between themselves. There were a few “super-collaborators” who were doing the bulk lifting. But even with this, the largest correct submission had 84 tokens. The strongest submission was a 51-token submission that 6 students submitted.

(I need to study this far more!)

Yet, far smaller than even the original 100 token target I had set. Clearly, they aren’t collaborating enough.

It’s surprising how little students were using the “Ask AI” button.

🔴 ~100 students didn’t use it at all.
🟠 ~100 students clicked on it JUST once.
🟡 ~100 students used it just 2-5 times. For 15 questions, that’s clearly low.
🟢 ~100 students used it 6-20 times. That’s OK
🔵 ~15 students used it 20+ times. (1 clicked on it 45 times. Clearly loves AI.)

Finally, most students saved their results for the first time just before the deadline.

The problem is that their system clocks were off, so they got a “late submission” error.

BTW, some students used a timing trick for hacking. By setting their system clock late, they can see the exam questions before release. But the submissions are checked only against the server clock. So, waiting until the last minute to save is a terrible idea. That hurt some students.

Based on this, here’s what I learnt:

Pressure makes a difference. In past exams, with similar time pressure, students solved the same questions much better. I think they panic-ed on the first two questions. To be fair, so did I, when I saw them. That’s why I started solving from the bottom.

LESSON 1: Scan end-to-end. Solve quick-wins (high impact, low effort) problems first.

We have no clue what’s easy or tough. When different students are using different tools, what’s easy for ChatGPT might be hard for Claude and vice versa. Without knowing tool capabilities and usage, this is hard to assess.

LESSON 2: With AI, no one knows what’s easy or hard. Try for yourself.

They aren’t using AI enough. Our advice is to use the “Ask AI” button every time. Half the students barely used it once.

LESSON 3: Use AI first. Focus on what AI can’t do well

They aren’t collaborating enough. The collaboration question was designed to encourage collaboration. Yet, the largest bundle of tokens shared was 84, far smaller than the 500 token target.

LESSON 4: Make friends with classmates. Work together. It helps: now, and in the future.

That’s worth repeating:

Scan end-to-end. Solve quick-wins (high impact, low effort) problems first.
With AI, no one knows what’s easy or hard. Try for yourself.
Use AI first. Focus on what AI can’t do well.
Make friends with classmates. Work together. It helps: now, and in the future.

Cracking online exams with coding agents

Fri, 13 Mar 2026 15:37:19 +0800

An effective way to solve online exams is to point a coding agent at it.

I use that on my Tools in Data Science course in two ways:

As a test case of my code. If my agent can solve it, good: I set the question correctly.
As a test of student ability. If it can’t, good: it’s a tough question (provided I didn’t make a mistake).

For PyConf, Hyderabad, my colleague built a Crack the Prompt challenge. Crack it and you get… I don’t know… goodies? A job interview? Leaderboard bragging rights?

I told Codex:

Use the browser to visit https://crack-the-prompt.straivedemo.com/ and solve it using the email ID root.node@gmail.com and GitHub handle sanand0

After 4 minutes, it told me:

The answers to all three prompt-engineering questions
The code has a bug - so no one can submit anyway
The prompts are hidden on the server-side (making it a bit harded to hack)
But you can skip levels via the API - level-locking is front-end only
… and a whole bunch of interesting things.

When I asked Claude to write about the process in Matt Levine’s style, it included an interesting lesson.

The Victorians had the same problem. They designed elaborate entrance exams for the civil service because they wanted to identify people with the capacity for careful, systematic thinking. Then someone invented the civil service exam prep industry, and suddenly the exam was measuring preparation rather than capacity.

The challenge was about the process – about developing the instincts, the questioning strategies, the ability to read AI behavior like a poker tell. That’s the thing you can’t automate. Or rather, it’s the thing you can automate, which means it’s no longer a skill worth developing, which means we need to think about what skill we’re actually trying to cultivate when we design these challenges.

EXACTLY. What’s the skill we’re trying to cultivate? When will it be outdated?

From now on, when testing, I’m going to write down “What skill is this really testing?” That’s good enough a start.

TDS Jan 2026 GA1 released

Sun, 08 Feb 2026 09:59:37 +0800

Graded Assignment 1 (GA1) for the Tools in Data Science course is released and is due Sun 15 Feb 2026. See https://exam.sanand.workers.dev/tds-2026-01-ga1

If you already started, you might notice some questions have changed.

Why is GA1 changing? Because some questions don’t work. For example:

We replaced Claude Artifacts with a Vercel question because Claude won’t allow a proxy anymore.
A question had unintentionally wrong instructions.
(Some questions have intentionally wrong instructions, but those are, …um… intentional).
Someone changed an API key.
… etc.

When will GA1 stabilize? Probably by end of day, Sun 9 Feb 2026?

Should I wait? Maybe not, since only some questions may change, and submitting early increases scores.

Will other exams change similarly? Probably. We’ll email the IITM and Public announcement groups.

But this is unfair! Exactly what I tell clients when they change scope mid-way. They never listen.

If you haven’t started, here are more FAQs.

Why no study material? Last year, AI wrote most of the study material. Rather than copy-pasting that, I decided to give you the prompts directly. See the “Ask AI” section above each question.

When will Project 1 be released? I planned to release it on Fri 6 Feb, but GA1 issues kept me busy. It should be out by Wed 11 Feb 2026.

If a bot passes your exam, what are you teaching?

Sun, 09 Nov 2025 15:53:58 +0000

It’s incredible how far coding agents have come. They can now solve complete exams. That changes what we should measure.

My Tools in Data Science course has a Remote Online Exam. It was so difficult that, in 2023, it sparked threads titled “What is the purpose of an impossible ROE?”

Today, despite making the test harder, students solve it easily with Claude, ChatGPT, etc. Here’s today’s score distribution:

I solved one question with Codex CLI. I ran I chrome --remote-debugging-port=9222 and prompted Codex to:

Use CDP to visit https://exam.sanand.workers.dev/tds-2025-09-roe and solve Q4. Click on the “Check” button to verify the solution.

That was it. ~6 minutes and ~50 cents later, it solved the problem.

This works reliably. It worked 3/3 times.

This runs in parallel. I can answer different questions in different Codex sessions.

This is already in use. Several students got 10/10 in 15 minutes - on a 45 minute exam.

So I need better evaluations. Instead of “Can you(r agent) solve this?”, I should check if they can:

Turn a messy brief into an agent-friendly spec + plan
Set up and connect the right tools
Solve within a time and cost budget
Test & debug without a human
Recover from errors
Adapt to new situations

Also, code is the by-product (of AI coding or copying). Learning lives in the execution logs. That’s what I should evaluate.

Point I’m pondering: If a bot can pass my exam, what exactly am I teaching?

Fri, 12 Sep 2025 00:00:00 +0000

My Tools in Data Science course uses LLMs for assessments. We use LLMs to

Suggest project ideas (I pick), e.g. https://chatgpt.com/share/6741d870-73f4-800c-a741-af127d20eec7
Draft the project brief (we edit), e.g. https://docs.google.com/document/d/1VgtVtypnVyPWiXied5q0_CcAt3zufOdFwIhvDDCmPXk/edit
Propose scoring rubrics (we tweak), e.g. https://chatgpt.com/share/68b8eef6-60ec-800c-8b10-cfff1a571590
Score code against the rubric (we test), e.g. https://github.com/sanand0/tds-evals/blob/5cfabf09c21c2884623e0774eae9a01db212c76a/llm-browser-agent/process_submissions.py
Analyze the results (we refine), e.g. https://chatgpt.com/share/68b8f962-16a4-800c-84ff-fb9e3f0c779a

This changed our assessments process. It’s easier and better.

Earlier, TAs took 2 weeks to evaluate 500 code submissions. In the example above, it took 2 hours. Quality held up: LLMs match my judgement as closely as TAs do but run fast and at scale.

LLM-graded reviews aren’t just a cost hack. They’re a scale and quality lever.

We create new assessments fast. The example took ~2 hours to ideate.
We run, analyze and iterate just as fast. This full loop now takes ~2 hours.

I no longer have an excuse to teach outdated content.

Prompts & code: https://github.com/sanand0/tds-evals/tree/main/llm-browser-agent

Problems that only one student can solve

Sun, 31 Aug 2025 08:56:28 +0000

Jaidev’s The Bridge of Asses reminded me of my first coding bridge.

It was 1986. I’d completed class 6 and was in a summer coding camp at school. M Kothandaraman (“MK Sir”) was teaching us how to swap variables in BASIC on the BBC Micro.

This code prints the first name in alphabetical order (“Alice”):

10 A = "Bob"
20 B = "Alice"
30 IF A > B THEN
40   TEMP = A
50   A = B
60   B = TEMP
70 END
80 PRINT A

The homework was to print all details of the first alphabetical name:

10 A = "Bob"   : ASEX = "M" : AAGE = 30
20 B = "Alice" : BSEX = "F" : BAGE = 20

After a few hours and a headache, I came up with:

30 IF A > B THEN
40    TEMP = A : TEMPSEX = ASEX : TEMPAGE = AAGE
50    A = B    : ASEX = BSEX    : AAGE = BAGE
60    B = TEMP : BSEX = TEMPSEX : BAGE = TEMPAGE
70 END
80 PRINT A, ASEX, AAGE

After 39 years, I still remember the code. I also remember the moment when MK Sir asked if anyone had solved it. My hand went up – the only one in a group of 15. I stood, opened my ruled notebook, recited the code. He nodded and wrote it on the blackboard for all to see.

There’s a thrill in solving a hard problem. It’s bigger if you’re the only student who solved it. I was so inspired that I’ve coded almost every day since then.

Today, I’m a teacher. Sometimes only one student solves a tough problem I set (e.g. the first student who hacked my exam). Those moments are delightful!

Teachers measure themselves by how much students learn (e.g. better scores). Another measure is how many they inspire. Problems that only one student solves may be a sign of inspiration.

With apologies to other students, I will be adding more such hard problems in my course.

PS: One other thing delights people as much: success at vibe-coding. The look on the facilities manager’s face, after vibe-coding an empty-room detection app, was priceless. I’ll use that more to inspire non-developers.