Tools in Data Science has a remote online exam (ROE). It has a tough reputation. We conducted one today. Here’s how today’s ROE unfolded. The TAs had created 13 questions and shared it with me yesterday. This morning, I tried solving them. At first glance, it looked scarily hard! But I just jumpted down a few questions, and found that five questions were trivial, i.e. I just used the “Ask AI” button to copy the question into ChatGPT and it gave me the answer. ...
An effective way to solve online exams is to point a coding agent at it. I use that on my Tools in Data Science course in two ways: As a test case of my code. If my agent can solve it, good: I set the question correctly. As a test of student ability. If it can’t, good: it’s a tough question (provided I didn’t make a mistake). For PyConf, Hyderabad, my colleague built a Crack the Prompt challenge. Crack it and you get… I don’t know… goodies? A job interview? Leaderboard bragging rights? ...
Graded Assignment 1 (GA1) for the Tools in Data Science course is released and is due Sun 15 Feb 2026. See https://exam.sanand.workers.dev/tds-2026-01-ga1 If you already started, you might notice some questions have changed. Why is GA1 changing? Because some questions don’t work. For example: We replaced Claude Artifacts with a Vercel question because Claude won’t allow a proxy anymore. A question had unintentionally wrong instructions. (Some questions have intentionally wrong instructions, but those are, …um… intentional). Someone changed an API key. … etc. When will GA1 stabilize? Probably by end of day, Sun 9 Feb 2026? ...
It’s incredible how far coding agents have come. They can now solve complete exams. That changes what we should measure. My Tools in Data Science course has a Remote Online Exam. It was so difficult that, in 2023, it sparked threads titled “What is the purpose of an impossible ROE?” Today, despite making the test harder, students solve it easily with Claude, ChatGPT, etc. Here’s today’s score distribution: ...
My Tools in Data Science course uses LLMs for assessments. We use LLMs to Suggest project ideas (I pick), e.g. https://chatgpt.com/share/6741d870-73f4-800c-a741-af127d20eec7 Draft the project brief (we edit), e.g. https://docs.google.com/document/d/1VgtVtypnVyPWiXied5q0_CcAt3zufOdFwIhvDDCmPXk/edit Propose scoring rubrics (we tweak), e.g. https://chatgpt.com/share/68b8eef6-60ec-800c-8b10-cfff1a571590 Score code against the rubric (we test), e.g. https://github.com/sanand0/tds-evals/blob/5cfabf09c21c2884623e0774eae9a01db212c76a/llm-browser-agent/process_submissions.py Analyze the results (we refine), e.g. https://chatgpt.com/share/68b8f962-16a4-800c-84ff-fb9e3f0c779a This changed our assessments process. It’s easier and better. Earlier, TAs took 2 weeks to evaluate 500 code submissions. In the example above, it took 2 hours. Quality held up: LLMs match my judgement as closely as TAs do but run fast and at scale. ...
Jaidev’s The Bridge of Asses reminded me of my first coding bridge. It was 1986. I’d completed class 6 and was in a summer coding camp at school. M Kothandaraman (“MK Sir”) was teaching us how to swap variables in BASIC on the BBC Micro. This code prints the first name in alphabetical order (“Alice”): 10 A = "Bob" 20 B = "Alice" 30 IF A > B THEN 40 TEMP = A 50 A = B 60 B = TEMP 70 END 80 PRINT A The homework was to print all details of the first alphabetical name: ...