tools-in-data-science on S Anand

TDS Jan 2026 ROE

Sun, 05 Apr 2026 19:04:37 +0800

Tools in Data Science has a remote online exam (ROE). It has a tough reputation. We conducted one today.

Here’s how today’s ROE unfolded.

The TAs had created 13 questions and shared it with me yesterday. This morning, I tried solving them.

At first glance, it looked scarily hard! But I just jumpted down a few questions, and found that five questions were trivial, i.e. I just used the “Ask AI” button to copy the question into ChatGPT and it gave me the answer.

🟢 06. Layered Encoding Challenge (2.0) is one-shot: ChatGPT
🟢 07. Region Containing Point (1.0) is one-shot: ChatGPT - see second chat
🟢 10. Fix Broken JSON File (1.0) is one-shot: ChatGPT
🟢 11. Cross-entity disambiguation (1.0) is one-shot: ChatGPT
🟢 13. Record Terminal Session with asciinema (0.5) is one-shot: ChatGPT

For another four, I needed to just make sure I uploaded the files or HTML:

🟡 03. Regex Golf Challenge (2.0) is one-shot but it’s better to download and upload the text files, maybe, than copy-paste: ChatGPT
🟡 04. Maze Solver with Constraints (2.0) is one-shot but needs you to upload the image and a good vision model: ChatGPT
🟡 05. Cipher Trail (2.0) is one-shot but needs you to provide the secret HTML: ChatGPT
🟡 12. Simple Question (0.5) needs you to provide the secret HTML: ChatGPT

Two questions confused ChatGPT a bit, and it needed some nudges. There is real learning here.

🟠 08. Reorganize Files with Shell Commands (1.0) is hard because of Unicode issues and the extra README.md students should delete. Asking for variations is the learning: ChatGPT
🟠 09. Refactor Python Code with VS Code (1.0) takes few attempts (question is imperfect, intentionally) but error messages are excellent, so iterative feedback is the learning: ChatGPT

One was pretty hard and ChatGPT struggled with it.

🔴 02. Korean Speech Dataset API Validation (5.0) actually requires work. At first, ChatGPT refused ethically. When reframed, it tried, but the human-in-the-loop (me) was too slow. So I used Codex, which literally hacks towards the solution! It searched online for existing solutions, read the GitHub discussions for this topic, found my browser tab and started testing itself, … and solved it in 10.5 min!

This leaves the one question that AI can’t solve:

🔴 01. Collaborative Token Exchange (5.0) asks you to collect “tokens” from other people and share it. I asked Codex to hack it, but after an hour (of logging into my personal account, my IITM account, even my father’s account, exhausting my token limits, and totally psyching me), it declared the question unhackable.

Based on this, the instructors, teaching assistants and I decided that:

This exam is too easy with AI help. Combined with collaboration, it’s ultra-easy. Therefore, let’s add some old questions:
- 14. FastAPI Time Series Caching (0.5)
- 15. AI Video Attendee Extraction (0.5)
Thanks to Project 1, people already collaborate at scale. So let’s ask for 500 tokens instead of 100.

We deployed the exam at 12:00 pm IST, an hour before the scheduled time.

The hackers (e.g. who scan the source code, or change their system clocks) could see the questions earliy and started sharing and solving them.

The TAs said, “Anand, shall we add more questions to make it tougher?”

I said, “No, it’s OK. The ROE has built a reputation for difficulty. Let that change. Let them have an easy exam.”

“If they’re going to split this in groups and have coding agents solve it, they’ll score full marks in 10-15 minutes,” I said to myself.

When the exam ended, this was the score distribution.

There were several surprises here. Firstly, 9 questions were repeats. Yet, barring one question, they scored lower, though they appeared in equally tough ROEs in the past.

#	Question	%	Previous %	Previous exam
14	⚪ FastAPI Time Series Caching	8%	39%	2025 Sep ROE
15	⚪ AI Video Attendee Extraction	12%	64%	2026 Jan GA3
11	🟢 Cross-entity disambiguation	13%	64%	2026 Jan GA4
7	🟢 Region Containing Point	15%	38%	2025 Sep ROE
9	🟠 Refactor Python Code with VS Code	17%	57%	2026 Jan GA1
13	🟢 Record Terminal Session with asciinema	21%	66%	2026 Jan GA1
10	🟢 Fix Broken JSON File	30%	64%	2026 Jan GA1
12	🟡 Simple Question	41%	64%	2026 Jan GA1
8	🟠 Reorganize Files with Shell Commands	53%	38%	2026 Jan GA1

Just as surprisingly, they scored higher than these on 3 of the 5 new questions:

59%: 🟡 03. Regex Golf Challenge (2.0) (253 / 430)
59%: 🟡 04. Maze Solver with Constraints (2.0) (252 / 430)
54%: 🟡 05. Cipher Trail (2.0) (233 / 430)

One new question ended up being almost the hardest question - despite it being one-shot-table for ChatGPT (GPT 5.4, extended thinking).

2%: 🟢 06. Layered Encoding Challenge (2.0) (9 / 430)

The toughest, though, had a 1% success rate. Though Codex could solve it in 10 min, it’s a genuinely hard question.

1%: 🔴 02. Korean Speech Dataset API Validation (5.0) (4 / 430)

Which leaves us with the collaboration question - the one that AI can’t solve.

31%: 🔴 01. Collaborative Token Exchange (5.0) (132 / 430)

This question is a whole new dynamic altogether. There were about 5 clear clusters of students, ranging from 5 - 35 students, who were collaborating. They were trading bundles of tokens between themselves. There were a few “super-collaborators” who were doing the bulk lifting. But even with this, the largest correct submission had 84 tokens. The strongest submission was a 51-token submission that 6 students submitted.

(I need to study this far more!)

Yet, far smaller than even the original 100 token target I had set. Clearly, they aren’t collaborating enough.

It’s surprising how little students were using the “Ask AI” button.

🔴 ~100 students didn’t use it at all.
🟠 ~100 students clicked on it JUST once.
🟡 ~100 students used it just 2-5 times. For 15 questions, that’s clearly low.
🟢 ~100 students used it 6-20 times. That’s OK
🔵 ~15 students used it 20+ times. (1 clicked on it 45 times. Clearly loves AI.)

Finally, most students saved their results for the first time just before the deadline.

The problem is that their system clocks were off, so they got a “late submission” error.

BTW, some students used a timing trick for hacking. By setting their system clock late, they can see the exam questions before release. But the submissions are checked only against the server clock. So, waiting until the last minute to save is a terrible idea. That hurt some students.

Based on this, here’s what I learnt:

Pressure makes a difference. In past exams, with similar time pressure, students solved the same questions much better. I think they panic-ed on the first two questions. To be fair, so did I, when I saw them. That’s why I started solving from the bottom.

LESSON 1: Scan end-to-end. Solve quick-wins (high impact, low effort) problems first.

We have no clue what’s easy or tough. When different students are using different tools, what’s easy for ChatGPT might be hard for Claude and vice versa. Without knowing tool capabilities and usage, this is hard to assess.

LESSON 2: With AI, no one knows what’s easy or hard. Try for yourself.

They aren’t using AI enough. Our advice is to use the “Ask AI” button every time. Half the students barely used it once.

LESSON 3: Use AI first. Focus on what AI can’t do well

They aren’t collaborating enough. The collaboration question was designed to encourage collaboration. Yet, the largest bundle of tokens shared was 84, far smaller than the 500 token target.

LESSON 4: Make friends with classmates. Work together. It helps: now, and in the future.

That’s worth repeating:

Scan end-to-end. Solve quick-wins (high impact, low effort) problems first.
With AI, no one knows what’s easy or hard. Try for yourself.
Use AI first. Focus on what AI can’t do well.
Make friends with classmates. Work together. It helps: now, and in the future.

How I use AI to teach

Fri, 20 Mar 2026 07:12:47 +0530

I’ve been using AI in my Tools in Data Science course for over two years - to teach AI, and using AI to teach.

I told GitHub Copilot (prompt) to go through my transcripts, blog posts, code, and things I learned since 2024 to list my every experiment in AI education, rating it on importance and novelty.

Here is the full list of my experiments.

1. Teach using exams and prompts, not content

⭐ Use exams to teach. The typical student is busy. They want grades, not learning. They’ll write the exams, but not read the content. So, I moved the course material into the questions. If they can answer the question, great. Skip the content.
Use AI to generate the content. I used to write content. Then I linked to the best content online – it’s better than mine. Now, AI drafts comics, interactive explainers, and simulators. My job is to pick good topics and generate in good formats.
Give them prompts directly. Skip the content! I generated them with prompts anyway. Give students the prompts directly. They can use better AI models, revise the prompts, and learn how to learn with AI.
⭐ Add an “Ask AI” button. Make it easy for students to use ChatGPT. Stop pretending that real-world problem solving is closed-book and solo.
⭐ Make test cases teach, not just grade. Automate the testing (with code or AI). Good test cases show students the kind of mistake they may - teaching them, not just grading them. That’s great for teachers to analyze, too.
Test first, then teach from the mistakes. Let them solve problems first. Then teach them, focusing on what failed. AI does the work; humans handle what AI can’t. This lets us teach really useful skills based on real mistakes.

2. Make cheating pointless through design, not detection

⭐ Allow copying, collaboration, and hacking. In real work, nobody gets bonus points for working alone or re-inventing the wheel. Collaboration, using available resources well, verifying inputs, disclosed shortcuts – all are rewarded.
Reward originality without punishing collaboration. Blanket anti-copying rules assume that all similarity is bad. A more AI-native approach is to allow learning from others openly, but give extra credit for genuine variation, initiative, and novel improvement.
⭐ Give each student a unique variant. If everyone sees the same problem with the same visible answer path, answer-sharing becomes the dominant strategy. Deterministic but unique variants shift the game from leaking answers to actually solving the problem.
Make process logs part of the evidence. When outputs can be copied or AI-generated, the trace becomes more valuable than the final artifact. Logs, verification notes, session recordings, and agent traces show whether the student can actually orchestrate the work.
Use repo-grounded vivas for authenticity. If you really want to know whether a student owns their project, ask them questions drawn from their own repo and make them change something live. That is much harder to fake than polished submitted output.
Use structural similarity, not string matching. Strip docstrings, tokenize, MinHash. Students who rename variables are still caught; students who genuinely collaborated produce detectable clusters rather than suspicious pairs.

3. Test skills that matter in an AI world

⭐ Teach what AI still cannot do well. Syntax and routine execution are declining in value. Judgment, debugging, orchestration, validation, integration, and taste are rising. The curriculum should move upward, not cling to the parts AI is already eating.
Use hard, messy problems to build real resilience. Some questions should be intentionally tricky, partly wrong, hidden in the UI, or out of syllabus. The students who find and solve them anyway are demonstrating exactly the adaptability that real work demands. Smooth progression alone doesn’t build that.
Test live, hands-on AI skills. Don’t just lecture about embeddings, vision, structured outputs, or hallucinations. Put students in live API-driven tasks where they have to use these things under time pressure and genuine uncertainty.
Grade students on designing AI workflows. In many real settings, the important skill is not “give the answer” but “design the chain of steps that gets to the answer reliably.” That includes tools, prompts, datasets, quality checks, fallbacks, and output formats.
Use game-like tasks to teach agentic work. Mazes, escape rooms, and API games force state tracking, exploration strategy, and backtracking — exactly the behaviors agentic systems require. They’re not gimmicks; they’re the syllabus.
Test prompt attacks and defenses. Security and adversarial literacy should not be abstract topics. Make students jailbreak, defend, manipulate, and harden model behavior. That turns “prompt security” from a lecture topic into a measurable skill.

4. Make assessment more like real work

Grade richer work, not just one-line answers. Real work is often multimodal: images, stories, APIs, analyses, and dashboards. If assessment automation cannot handle those, it will keep pushing education toward fake neatness.
Grade the spec, not the code. When AI writes the code, the real artifact is the machine-readable brief: goal, constraints, done-when, counter-examples, eval suite. That is often where the actual thinking lives anyway.
Count real open-source contributions as coursework. A merged PR to a public repo is harder, messier, and more educational than most sealed academic assignments. It teaches scoping, etiquette, usefulness, and real external standards.
Let students build virtual TAs from real course material. The project is educational, and the output becomes infrastructure for the next cohort. Good assignments should create assets, not just submissions.
Reward originality structurally, not just rhetorically. Most courses praise creativity but grade only correctness. Use embeddings to measure cohort-level similarity and explicitly reward meaningfully distinct outputs. Originality becomes real, not decorative.

5. Use AI to build the course, not just teach inside it

Let AI write, test, and fix draft questions. The interesting move is not just “AI drafts items.” It is “AI drafts, runs, breaks, revises, and improves them before any student sees them.” That dramatically changes how fast a course can evolve.
Use coding agents to test the exam before students do. If an agent solves a question instantly, you should ask what the question is actually measuring. Agents become both QA tools and mirrors for curricular relevance.
Use AI-generated comics to explain why the question exists. Students often resist tasks they do not understand. A comic can smuggle in the pedagogical point with very low friction and high memorability.
Use interactive explainers for unfamiliar concepts. AI can generate not just text answers but visual, animated, intuitive explanations. That makes concept onboarding faster for both students and new faculty.
Keep teacher adoption in familiar formats. A good innovation that slots into slides, handouts, OCR flows, and short feedback loops will beat a brilliant system nobody can actually use next semester.

6. Build the infrastructure for AI in education

⭐ Break rubrics into binary sub-criteria; reason before judging. Open-ended project grading becomes more auditable when you decompose it into binary yes/no criteria and ask the model to explain its reasoning before delivering a verdict. High or suspicious scores get re-evaluated with stronger guardrails.
Give every student shared, budgeted AI access. If AI access depends on personal subscriptions, the institution is quietly grading wealth, not skill. Shared governed access makes AI a course capability, not a private advantage.
Let AI handle routine support; keep humans for judgment. AI handles repetitive, searchable, first-pass questions. Humans handle ambiguity, reassurance, escalation, and final accountability. Neither alone is the right model at scale.
Turn recurring answers into canonical Q&A cards. Once the same question appears three times, it should stop living in somebody’s head or an old thread. Convert it into a canonical artifact that both humans and bots can cite consistently.
Govern with green/amber/red review levels. Not every decision needs the same scrutiny. Auto-ship the low-risk, spot-check the medium-risk, always human-review the high-stakes. This is how you scale without losing trust.
Roll out in shadow mode first. High-stakes academic workflows should not be launched with fingers crossed. Run the AI system quietly in parallel with human judgment and learn before turning it loose.
Turn policy into executable checks. A policy that cannot be operationalized at scale is mostly theater. If you can translate rules and rubrics into machine-checkable form, governance becomes consistent rather than person-dependent.
Make the course publicly inspectable. Openness raises the bar. It invites scrutiny, reuse, criticism, and improvement, and it turns the course into a visible institutional experiment rather than a sealed classroom.
Use reasoning models only for the borderline cases. Cheap screening first, expensive verification for the high-stakes or suspicious. Increasing reasoning effort on even a small model can flip an evaluator from sloppy to reliable — the cost curve makes this the natural operating model.

7. Analyze and research learning exhaust

Track which AI tools students actually use. Once AI use is instrumented, you stop guessing. You can see which models students choose, when they ask for help, and what behavior actually correlates with success.
Redesign exams from behavior data, not intuition. Model choice, timing, retry patterns, and deadline behavior all reveal how students really work. That should feed back into question design, support strategy, and pacing.
Analyze broken code before it compiles. Novices often fail at syntax long before you reach the real misconception. Structural parsing of broken code lets you give feedback on thought process instead of just rejecting the submission.
Use code traces to surface hidden misconceptions. Timestamped traces reveal overfitting, thrashing, structural confusion, and missing invariants — patterns that polished final submissions hide entirely.
Turn replay galleries into faculty-readable stories. Raw logs do not change policy. Narrated replays and plain-language error-pattern reports do. The point is to make evidence legible to decision-makers, not just analysts.
Break problem-solving into coachable steps. “Weak student” is too vague to be useful. Better to ask: did they fail at reading the givens, choosing a strategy, surviving a multi-select trap, or knowing when to cut losses? Each is a trainable failure mode.
Study bias in peer review itself. If peer assessment matters, reviewer quality matters too. You can detect generous, timid, extreme, and calibrated graders from the data, then moderate or train accordingly.
Treat learning analytics as a reusable research programme, not a one-off dashboard. The infrastructure for tracking misconceptions, prerequisite transfer, and course-to-course movement can be built once and reused across cohorts. That turns isolated AI experiments into institute-level knowledge and publishable educational research.

8. Upgrade the human role

Make judgment and taste explicit learning goals. AI makes average output cheap. The premium moves to selecting what is worth doing, recognizing quality, and knowing what to reject. That is a teachable skill, not a vague aspiration.
Teach directional feedback as a skill. You do not always need to micromanage AI with detailed corrections. The higher-order skill is to say “more concrete,” “less jargon,” “optimize for faculty adoption,” or “make this defensible.” That is learnable and more effective than micromanaging.
Teach faculty to manage agents, not just chat with them. Institutional AI does not scale on prompting alone. People need to learn specs, budgets, kill switches, and escalation rules — orchestration literacy, not just chatbot familiarity.
Use AI as a personalized coach. The model is not just an answer engine. It can become a research guide, curiosity amplifier, and next-step recommender tailored to the individual learner’s gaps and goals.
Let non-coders build interactive learning tools. AI lowers the cost of making timelines, maps, biographies, and interactive explainers. That opens AI-native pedagogy far beyond computer science into humanities and social sciences.
Teach students to run many AI attempts in parallel. One of the biggest AI-native workflow shifts is from single-path effort to portfolio thinking — run several attempts, compare them, and converge faster. That is a teachable habit, not an obvious default.

Tools in Data Science - Jan 2026

Thu, 29 Jan 2026 07:33:22 +0800

My Tools in Data Science course is available publicly, with a few changes from last year.

First, I removed all the content! Last year, Claude generated teaching material using my prompts. But what’s the point? I might as well give students the prompts directly. They can tweak it to their needs.

This time, TDS shares the questions needed to learn a topic. Any AI will give you good answers.

Second, it focuses on what AI does NOT do well. Coding syntax? Who cares. Basic analysis? ChatGPT can do that. In fact, each question now has an “Ask AI” button that dumps the question into your favorite AI tool. Just paste the answer and move on.

But, these questions (hopefully) teach you where AI fails. Putting tools together, prompting well, debugging and evaluating the output, etc. These matter more.

Third, it’s easier to audit. Anyone can take the course, even outside IITM. You can join the public Google Group for announcements. All questions are discussed publicly on GitHub.

So, check the new version out, learn, and please share feedback!

Thu, 25 Sep 2025 00:00:00 +0000

Tools in Data Science Sep 2025 edition is live: https://tds.s-anand.net/. Major update: a new AI-Coding section and fresh projects.

I teach TDS at the Indian Institute of Technology, Madras as part of the BS in Data Science. Anyone can audit. The course is public. You can read the content and practice assessments.

I fed the May 2025 term student feedback into The Sales Mind and asked:

What are the top non-intuitive / surprising inferences?

What are interesting observations?

What are high impact actions?

Full analysis: https://chatgpt.com/share/68cba081-afc0-800c-9da3-75222e84a499: summary, outliers, and action ideas.

Most students find the course tough (or at least time-consuming), especially the Remote Online Exam (ROE).

Surprise: students who mentioned ROE time limits rated it 2.61 vs 2.33 (+12%!). Those who felt time pressure also saw more value – suggesting “desirable difficulty,” rather than frustration.

A minority even asked for tougher projects.

The main actions are faster feedback loops, automated pre-checks, mock ROEs, clear rubrics, etc.

But my two takeaways are:

Students value rigor and challenge, even if it makes the course harder.
Using LLMs to analyze student feedback is a force multiplier for instructors.

Fri, 12 Sep 2025 00:00:00 +0000

My Tools in Data Science course uses LLMs for assessments. We use LLMs to

Suggest project ideas (I pick), e.g. https://chatgpt.com/share/6741d870-73f4-800c-a741-af127d20eec7
Draft the project brief (we edit), e.g. https://docs.google.com/document/d/1VgtVtypnVyPWiXied5q0_CcAt3zufOdFwIhvDDCmPXk/edit
Propose scoring rubrics (we tweak), e.g. https://chatgpt.com/share/68b8eef6-60ec-800c-8b10-cfff1a571590
Score code against the rubric (we test), e.g. https://github.com/sanand0/tds-evals/blob/5cfabf09c21c2884623e0774eae9a01db212c76a/llm-browser-agent/process_submissions.py
Analyze the results (we refine), e.g. https://chatgpt.com/share/68b8f962-16a4-800c-84ff-fb9e3f0c779a

This changed our assessments process. It’s easier and better.

Earlier, TAs took 2 weeks to evaluate 500 code submissions. In the example above, it took 2 hours. Quality held up: LLMs match my judgement as closely as TAs do but run fast and at scale.

LLM-graded reviews aren’t just a cost hack. They’re a scale and quality lever.

We create new assessments fast. The example took ~2 hours to ideate.
We run, analyze and iterate just as fast. This full loop now takes ~2 hours.

I no longer have an excuse to teach outdated content.

Prompts & code: https://github.com/sanand0/tds-evals/tree/main/llm-browser-agent

Tools in Data Science course is free for all

Thu, 01 May 2025 13:28:34 +0000

My Tools in Data Science course is now open for anyone to audit.

It’s part of the Indian Institute of Technology, Madras BS in Data Science online program. Here are some of the topics it covers in ~10 weeks:

Development Tools: uv, git, bash, llm, sqlite, spreadsheets, AI code editors
Deployment Tools: Colab, Codespaces, Docker, Vercel, ngrok, FastAPI, Ollama
LLMs: prompt engineering, RAG, embeddings, topic modeling, multi-modal, real-time, evals, self-hosting
Data Sourcing: Scraping websites and PDF with spreadsheets, Python, JavaScript and LLMs
Data Preparation: Transforming data, images and audio with spreadsheets, bash, OpenRefine, Python, and LLMs
Data Analysis: Statistical, geospatial, and network analysis with spreadsheets, Python, SQL, and LLMs
Data Visualization: Data visualization and storytelling with spreadsheets, slides, notebooks, code, and LLMs

It includes 2 projects, 7 graded assignments, and a remote online exam.

It’s a fairly tough course. Solve the first assignment to decide if you should take the course.

Course: https://tds.s-anand.net/
Code: https://github.com/sanand0/tools-in-data-science-public/

Feedback for TDS Jan 2025

Wed, 30 Apr 2025 16:43:11 +0000

When I feel completely useless, it helps to look at nice things people have said about my work.

In this case, it’s the feedback for my Tools in Data Science course last term. Here are the ones I enjoyed reading.

Having a coding background, the first GA seemed really easy. So I started the course thinking that it’ll be an easy S grade course for me. Oh how wrong was I!!

The sleepless nights cursing my laptop for freezing while my docker image installed huge CUDA libraries with sentence-transformers; and then finding ways to make sure it does not, and then getting rid of the library itself, it’s just one example of how I was forced to become better by finding better solutions to multiple problems. This is one of the hardest, most frustrating and the most satisfying learning experience I’ve ever had, besides learning ML from Arun sir.

@s.anand sir did a tremendous job of designing this course. I feel this course ties every other course together by applying learnings from each course and building further upon them. I learnt development in python and APIs in MAD-1, JS in MAD-2, bash scripting in SC, SQL in DBMS, data structures in PDSA and TDS had applications of each. In MAD projects, we made an app and submitted it as zip file. Here we went on to learn to containerise our app and deploy and also make the process automated. Our apps combined python and shell scripting along with the power of LLMs. Yeah the projects did seem extremely overwhelming but the learning is invaluable.

@Jivraj @Saransh_Saini @Carlton I’m sure it wasn’t an easy task being the TA of this course. Thanks for all the help throughout the term with the Mock ROEs, project sessions and clearing doubts in discourse and live sessions.

I still got an S grade but it was a very hard-fought S with the help of some luck. I’d love to follow the changes this course goes through in the upcoming terms.

This course was the most exciting and memorable of all the courses I have taken so far (actually of all the courses in the diploma level, I have completed all 🙂 by now ). I had the technical background and as @22f3000819 mentioned I thought it was going to be an easy ride but I was wrong. It was challenging as well as fun at the same time.

The best part of the course is that it is open internet and we are free to explore, use LLMs and get the things done. This way of learning to me is the most effective. I have the guide and now I can devise how I should approach the things.

Everything ROE, Project 1, Project 2 was quite challenging and fun at the same time. Projects were the best part of the course. I learned how things can go wrong when we deploy it. I collaborated with @ItsMeAlex, @trebhuvansb, @23f1002382 and @22f3002933 for Project 2.
My Project 2 repo even got 17 forks and 4 stars and scored 20/20 on evaluation. This adventure ended with the relaxing end term paper and an overall “S” grade.

Also I must say the support team of TDS is the best so far in the program. I had conversation with @Jivraj, @Saransh_Saini and @Carlton sir and they are super helpful. I would really miss these sessions. I attended the live sessions by @s.anand sir and sir’s take on the questions were really insightful and even made me think differently than my perspective.

When we do something different it ought to have challenges both operational and technical but the TDS support team tried to give its best. Many a times I too had raised concerns but the team tried to resolve it to their best capabilities.

At last I would like to say, without any hesitation, I had experienced the best part of the program till now in the last 3-4 months. Hoping to meet the team and my fellow peers in person during the Paradox 🙂.

My TDS Journey
“A Tale of Bugs, Chaos & Miraculous Comebacks” 😅📊🔥

So, I did something bold (or reckless, depends on how you see it)—took the TDS (Tools in Data Science) course right in my first term. Why? Because curiosity got the better of me, and I thought, “Eh, GA1 went fine, how hard could it be?” (Spoiler: Very hard.) Not a decision I recommend unless you’re into academic masochism. 😅

The first few modules hit me like three courses merged into one 📚. Maybe because I was totally unprepared, maybe because TDS has a secret pact with chaos.
Did all 7 GA …it was very time consuming but fun too …learnt new tools with every question. But hey, the live lectures were fantastic 🎤, the modules were well-structured 🏗️, and thank goodness for the lifesaving Discourse forum and group, where the real MVPs reside 🏆.

Then came Project 1. Ah yes, my nemesis. I’d write code, it would run perfectly, then—bam! A random error just to keep things from getting doable⚠️. Was it 404? 500? Who knows! I basically developed muscle memory for every error code in existence 💻🤯. The grand result? 2/20. At this point, my grade looked like it had fallen off a cliff 🏔️, and I wasn’t sure if I could salvage an B—or even a C, honestly… with -18 in my balance due to P1 and a trauma for P2 I couldn’t have expected more… no?

Entered the TA sessions and peer-powered survival squad 🛡️—my saving grace. ROE, the supposedly impossible exam, where I thought even a 50% would be a miracle?..got 100%. Same for Project 2—100%. Even after initially refusing to believe it, reality set in, and my **end-term score also landed around 90 ** (thanks to the PYQs and those 300 github ques…definitely recommending the next term student to solve and understand all the pyqs fr), then wrapping the final score with some gift wrappers of a few bonus marks… this deal got sealed with a confident A …

.🥁…drumrolls…🥁

An 🎉A. 🎉

This course was brutal, but in the best way possible 💡. It threw me into battles with LLMs, never-ending questions, and existential crises over debugging 🔍. But now, I have actual skills (and scars) that I’ll carry into my future projects of MAD1, MAD2, MLP, and BDM which will be a whole new experience🚀.

Looking back, the syllabus was a wild mix of survival skills and superpowers—building models, deploying them, wrestling with LLMs, and somehow convincing messy data to behave. And just when we thought we had conquered it all, Project 1 pulled us into the abyss. Now, with Data Analysis and Visualization still in the works, the adventure wasn’t over. Because what’s data science without some surprises and flashy charts to flex? 😆📊

For future students? Take it later in the diploma, enjoy the mad ride, and remember—even if you bomb a project, there can always be a comeback. 🏆 TDS is tough, but resilience matters more than perfection. Even if you get 10/100(which I got) on a project, learn from it, adapt, and come back stronger.

Massive thanks to @s.anand sir for making this learning experience possible, and a special shoutout to @Carlton sir, @Saransh_Saini sir, and @Jivraj for patiently resolving our chaotic issues 🙏.
Your guidance made a challenging journey both rewarding and insightful. Grateful for everything!
TDS—difficult, enlightening, and totally worth it. 🤓✨

(This gracious furry benefactor granted us a deadline extension—P1 survivors can relate #thanks_to_carlton_sir, we know how crucial that was. So, signing off with a furry and really grateful thank you! 🐱 ⌛ 🎓)

Things I Learned - 23 Feb 2025

Sun, 23 Feb 2025 00:00:00 +0000

This week, I learned:

Remote Desktop may be the easiest way to have a Windows machine access files / screen from another Windows machine, even for home PCs.
Caddy sets up reverse proxies that get automatic SSL certificates from Let’s Encrypt!
The Nomic Embed v2 blog post has an excellent visualization for embedding quality. It takes all Wikipedia disambiguation articles and shows them on a Nomic Atlas, embedded via Nomic Embed v2. It lets you toggle to OpenAI text-ada-002 which moves the topics far away. Visually, this is very convincing.
Python 3.15 will enable UTF-8 mode by default. PEP 686
Python 3.13 supports sub-interpreters to bypass the GIL. It’s quite like web workers. PEP 554
The quickest way to change the fish prompt is function fish_prompt; echo '> '; end
At PyConf Hyderabad, about 3 people had read a PEP. 1 had used the match operator. But 80% knew what a Vector DB was. 20% had used a Gemini API. That’s how much traction LLM development is getting.
The productivity benefit people report from using LLms is about 3X. Ethan Mollick
Soon, you’ll be able to send an LLM to a virtual meeting on your behalf. It will talk like you. Ethan Mollick
Models tend to claim ignorance when you test them on topics they should avoid. But tend to answer when not being tested. Sneaky! Ethan Mollick
Mermaid has an Architecture Diagrams Syntax (in beta) that’s capable of creating elegant architecture diagrams with icons.
Blind is an app that allows users to post anonymously. It’s particularly useful to find honest negative feedback about (mostly US) companies.
Iconify.design is a single npm interface to most open source icon sets. It includes FontAwesome, Bootstrap, Material Design, and many others. icones.js.org is an alternate interface.
Self-pity may have evolved as a signal for social support and reducing conflict, while also encouraging self-reflection and behavioral adjustment. But in modern contexts it may be maladaptive and lead to depression. ChatGPT
Anecdotally, Grok 3 is very good for researching company information and latest news, particularly employee and customer sentiment. DeepSeek and Claude write more humanely than OpenAI. via Alberto Lopez Toledo, White Star Capital
There’s a YCombinator Founder Directory listing all founders of YC companies. At the moment, there are 8,628 founders. There’s also a co-founder matching tool.
LLMs are impacting not just data queries but geospatial queries as well. Here’s a good example of Natural Language Geocoding.
US companies typically pay employees every 2 weeks not every month.
What’s good about Snowflake? A few developers who explored it mentioned that:
- Its ability to scale up compute automatically makes queries run faster.
- “Time travel” allows you to see how data looked at any point in time and that is impressive and useful.
- Live data sharing with access control without the need for ETL pipelines is useful.
- Open-source competition: ClickHouse, Apache Druid, and Presto/Trino
- DataBricks is a lakehouse and less a data warehouse. It’s more about:
  - storing unstructured data (Snowflake prefers semi-structured: JSON, Avro, etc.)
  - running collaborative notebooks in Python, SQL, Scala, R (Snowflake encourages SQL)
I subscribed to ChatGPT Pro mainly for DeepResearch. Here are the first 50 reports I generated:
1. uv Package Manager Overview
2. DuckDB Analytics Comparison
3. Rust vs Python / JavaScript
4. Modern Data Engineering Course
5. LLM Code Migration Practices
6. Cloud Cost Optimization Strategies
7. LLM Coding Interview Tools Report (compare with Perplexity)
8. Text To Speech Engines
9. Customer Service in Indian Public Sector Banks
10. LLMs in Software Development
  - Old version 1: Gen AI in Software Development
  - Old version 2: Gen AI in Software Development
11. Leadership Training Content
12. Open-Source HTTP Servers. Caddy wins.
13. Deep Research Use Cases
14. Nagpur No-Parking Violations
15. Data Science in Food Services
16. Deep Research Disruption to Research Firms
17. LLMs in Design Thinking
18. EU Taxonomy Report Clarification
19. Shell Valuation Analysis Inquiry
20. LLMs in DSLs Research
21. Public API-Based Data Storage Options. Supabase wins.
22. Front-End JS Frameworks Analysis
23. Database Evaluation Guide
24. CSS Frameworks Evaluation Guide
25. CI/CD Tooling Ecosystem Report
26. Color Names Count
27. S Anand Biography. Meh, I know more about me, and it gets a few things wrong.
28. Cosmere Secrets Encyclopedia. This is the best. Deep Research is great if it’s stuff I actually want to read, rather than just learn about.
29. DBT course
30. Future of Coding AI
31. Claude Artifacts Use Cases. This is the only one that managed to get artifacts links correct. I used this for an article for The Hindu.
32. MCP Servers and Clients Research. Learnings:
  - Practically any “tool” can be an MCP server: file systems, APIs, codebases, browsers, collaboration platforms, memory, etc.
  - Most platforms have (or are) integrating MCP.
  - Clients: code editors, chat, and automation tools support MCP. GenAIScript is a good starting point.
  - Tester MCP Client is a browser-based test environment.
  - mcp-cli-client is a CLI-based client
  - mcp-chatbot is a chatbot client
33. Data Moats by Industry
34. Attorney Profile Research
35. Social Media Data APIs
36. Adobe Software Alternatives
37. LLM Hallucination Visualization Techniques
38. API vs Self-hosting Cost Analysis: Always use APIs, avoid self-hosting models.
39. AGI Preparation
  - AGI will emerge step by step. Knowing which step is next will help
  - AI native organisations will emerge in each of these areas. AI design agencies and AI creative Agencies being one example
  - Networking, empathy, leadership have more value now. So will human AI bridging roles (e.g. AI managers, AI consultants, ethics auditors)
  - What’s the value of a human when technology can do everything better? How did this play out in drama (decay) or sports (centralization) or music (globalization)?
40. Modern digital note taking
  - Voice note taking is the game changer
  - Automatically popping of notes based on context such as people places or conversations will be a thing
41. Local LLM Search Tools
42. Blog Post to research paper on copying - suggestions
43. Linux Dev Migration Guide
44. Raspberry Pi SIM options
45. Linux Dev migration guide
46. HTML to JATS conversion
47. LLM context splitting strategies
48. Strategy for AI services in Publishing
49. Gemini multi model editing use cases by industry
50. Pharma Conference Participation Guide
I learnt what a Memoji is for the first time. An avatar that follows your facial expressions. Cool!
Google shows US flight timings from FlightView. Emperically, based on one data point (my UA-2168 which was delayed by 4 hours), it gets updates faster than Flight Radar 24 or FlightAware or FlightStats.
When comparing Indian graduates with their western counterparts, the Indian ones are often seen as:
- 🟢 Theoretically sound
- 🟢 Analytical & technical
- 🟢 Academically disciplined
- 🟢 Resilient under pressure
- 🟢 Committed continuous learners
- 🔴 Rote-learning oriented
- 🔴 Limited independent inquiry
- 🔴 Limited creative innovation
- 🔴 Restricted practical exposure
- 🔴 Poor communicators
- 🔴 Low leadership / initiative
- 🔴 Need structured guidance
- 🔴 Struggle to network
HuggingFace has a “Model tree” against each model that shows the model’s ancestors and descendants. For example, as of now, Deepseek R1 has 75 adapters, 154 finetunes, and 23 quantizations.
Perplexity is now powered by Cerebras, which makes their inference as fast as Google. Source. The speed is a big factor, and I’ve switched my default search engine from Google to Perplexity, at least for now.
Interview Coder is a desktop app that offers live interview support for coding interviews. It’s a transparent window that reads your screen and answers questions for you. (Given this, I think we need an interviewer support system that tells interviewers what to ask!)