tools-in-data-science 9

20 May 2026 Suhani01 I received a mail mentioning, “If you have already completed the course in the past 1 year, we encourage you to share your feedback about your experiences and what worked for you in TDS in the same Discourse thread.” So, here goes: I had taken TDS in May 2025 term ad I did not know it would be so difficult, because I had not read the guidelines before selecting it. And by the time I read the guidelines, I had already paid the registration fee. 🥲 Although I ended up with ‘C’ grade, this was the most enjoyable and unpredictable course out of all diploma courses. A lot of students might disagree with me calling this course ‘enjoyable’, but this is the only course which keeps you on your toes and challenges you every single day. You can make the maximum number of friends in TDS, because this course is not about being intelligent. It’s about the skill of ‘jugaad’ (being resourceful and smart enough to work around problems, not necessarily solve them). TDS teaches you stuff which you can never get out of conventional textbooks, like figuring out what resources to use, how to read material selectively, and how to use the system to beat the system. Most importantly, TDS teaches how to deal with real-life corporate and industry scenarios, which are also as unpredictable, and have no defined rules or syllabus. You’ll never get a textbook or steps to follow, you will have to read the documentation, watch YouTube videos, and discuss with friends to get to the solution yourself. Moreover, the satisfaction of completing a GA after hours of effort is unmatched! 😃 Most importantly, TDS has taught me consistency and resourcefulness, and given me a weird sense of confidence. Now whenever I get something difficult, like an undoable project or assignment, I just think that if I can complete TDS, then there’s practically nothing that I can’t do 😁! ...

Tools in Data Science has a remote online exam (ROE). It has a tough reputation. We conducted one today. Here’s how today’s ROE unfolded. The TAs had created 13 questions and shared it with me yesterday. This morning, I tried solving them. At first glance, it looked scarily hard! But I just jumpted down a few questions, and found that five questions were trivial, i.e. I just used the “Ask AI” button to copy the question into ChatGPT and it gave me the answer. ...

I’ve been using AI in my Tools in Data Science course for over two years - to teach AI, and using AI to teach. I told GitHub Copilot (prompt) to go through my transcripts, blog posts, code, and things I learned since 2024 to list my every experiment in AI education, rating it on importance and novelty. Here is the full list of my experiments. 1. Teach using exams and prompts, not content ⭐ Use exams to teach. The typical student is busy. They want grades, not learning. They’ll write the exams, but not read the content. So, I moved the course material into the questions. If they can answer the question, great. Skip the content. Use AI to generate the content. I used to write content. Then I linked to the best content online – it’s better than mine. Now, AI drafts comics, interactive explainers, and simulators. My job is to pick good topics and generate in good formats. Give them prompts directly. Skip the content! I generated them with prompts anyway. Give students the prompts directly. They can use better AI models, revise the prompts, and learn how to learn with AI. ⭐ Add an “Ask AI” button. Make it easy for students to use ChatGPT. Stop pretending that real-world problem solving is closed-book and solo. ⭐ Make test cases teach, not just grade. Automate the testing (with code or AI). Good test cases show students the kind of mistake they may - teaching them, not just grading them. That’s great for teachers to analyze, too. Test first, then teach from the mistakes. Let them solve problems first. Then teach them, focusing on what failed. AI does the work; humans handle what AI can’t. This lets us teach really useful skills based on real mistakes. 2. Make cheating pointless through design, not detection ...

My Tools in Data Science course is available publicly, with a few changes from last year. First, I removed all the content! Last year, Claude generated teaching material using my prompts. But what’s the point? I might as well give students the prompts directly. They can tweak it to their needs. This time, TDS shares the questions needed to learn a topic. Any AI will give you good answers. Second, it focuses on what AI does NOT do well. Coding syntax? Who cares. Basic analysis? ChatGPT can do that. In fact, each question now has an “Ask AI” button that dumps the question into your favorite AI tool. Just paste the answer and move on. ...

Tools in Data Science Sep 2025 edition is live: https://tds.s-anand.net/. Major update: a new AI-Coding section and fresh projects. I teach TDS at the Indian Institute of Technology, Madras as part of the BS in Data Science. Anyone can audit. The course is public. You can read the content and practice assessments. I fed the May 2025 term student feedback into The Sales Mind and asked: What are the top non-intuitive / surprising inferences? What are interesting observations? What are high impact actions? Full analysis: https://chatgpt.com/share/68cba081-afc0-800c-9da3-75222e84a499: summary, outliers, and action ideas. ...

My Tools in Data Science course uses LLMs for assessments. We use LLMs to Suggest project ideas (I pick), e.g. https://chatgpt.com/share/6741d870-73f4-800c-a741-af127d20eec7 Draft the project brief (we edit), e.g. https://docs.google.com/document/d/1VgtVtypnVyPWiXied5q0_CcAt3zufOdFwIhvDDCmPXk/edit Propose scoring rubrics (we tweak), e.g. https://chatgpt.com/share/68b8eef6-60ec-800c-8b10-cfff1a571590 Score code against the rubric (we test), e.g. https://github.com/sanand0/tds-evals/blob/5cfabf09c21c2884623e0774eae9a01db212c76a/llm-browser-agent/process_submissions.py Analyze the results (we refine), e.g. https://chatgpt.com/share/68b8f962-16a4-800c-84ff-fb9e3f0c779a This changed our assessments process. It’s easier and better. Earlier, TAs took 2 weeks to evaluate 500 code submissions. In the example above, it took 2 hours. Quality held up: LLMs match my judgement as closely as TAs do but run fast and at scale. ...

My Tools in Data Science course is now open for anyone to audit. It’s part of the Indian Institute of Technology, Madras BS in Data Science online program. Here are some of the topics it covers in ~10 weeks: Development Tools: uv, git, bash, llm, sqlite, spreadsheets, AI code editors Deployment Tools: Colab, Codespaces, Docker, Vercel, ngrok, FastAPI, Ollama LLMs: prompt engineering, RAG, embeddings, topic modeling, multi-modal, real-time, evals, self-hosting Data Sourcing: Scraping websites and PDF with spreadsheets, Python, JavaScript and LLMs Data Preparation: Transforming data, images and audio with spreadsheets, bash, OpenRefine, Python, and LLMs Data Analysis: Statistical, geospatial, and network analysis with spreadsheets, Python, SQL, and LLMs Data Visualization: Data visualization and storytelling with spreadsheets, slides, notebooks, code, and LLMs ...

When I feel completely useless, it helps to look at nice things people have said about my work. In this case, it’s the feedback for my Tools in Data Science course last term. Here are the ones I enjoyed reading. Having a coding background, the first GA seemed really easy. So I started the course thinking that it’ll be an easy S grade course for me. Oh how wrong was I!! The sleepless nights cursing my laptop for freezing while my docker image installed huge CUDA libraries with sentence-transformers; and then finding ways to make sure it does not, and then getting rid of the library itself, it’s just one example of how I was forced to become better by finding better solutions to multiple problems. This is one of the hardest, most frustrating and the most satisfying learning experience I’ve ever had, besides learning ML from Arun sir. ...

This week, I learned: Remote Desktop may be the easiest way to have a Windows machine access files / screen from another Windows machine, even for home PCs. Caddy sets up reverse proxies that get automatic SSL certificates from Let’s Encrypt! The Nomic Embed v2 blog post has an excellent visualization for embedding quality. It takes all Wikipedia disambiguation articles and shows them on a Nomic Atlas, embedded via Nomic Embed v2. It lets you toggle to OpenAI text-ada-002 which moves the topics far away. Visually, this is very convincing. Python 3.15 will enable UTF-8 mode by default. PEP 686 Python 3.13 supports sub-interpreters to bypass the GIL. It’s quite like web workers. PEP 554 The quickest way to change the fish prompt is function fish_prompt; echo '> '; end At PyConf Hyderabad, about 3 people had read a PEP. 1 had used the match operator. But 80% knew what a Vector DB was. 20% had used a Gemini API. That’s how much traction LLM development is getting. The productivity benefit people report from using LLms is about 3X. Ethan Mollick Soon, you’ll be able to send an LLM to a virtual meeting on your behalf. It will talk like you. Ethan Mollick Models tend to claim ignorance when you test them on topics they should avoid. But tend to answer when not being tested. Sneaky! Ethan Mollick Mermaid has an Architecture Diagrams Syntax (in beta) that’s capable of creating elegant architecture diagrams with icons. Blind is an app that allows users to post anonymously. It’s particularly useful to find honest negative feedback about (mostly US) companies. Iconify.design is a single npm interface to most open source icon sets. It includes FontAwesome, Bootstrap, Material Design, and many others. icones.js.org is an alternate interface. Self-pity may have evolved as a signal for social support and reducing conflict, while also encouraging self-reflection and behavioral adjustment. But in modern contexts it may be maladaptive and lead to depression. ChatGPT Anecdotally, Grok 3 is very good for researching company information and latest news, particularly employee and customer sentiment. DeepSeek and Claude write more humanely than OpenAI. via Alberto Lopez Toledo, White Star Capital There’s a YCombinator Founder Directory listing all founders of YC companies. At the moment, there are 8,628 founders. There’s also a co-founder matching tool. LLMs are impacting not just data queries but geospatial queries as well. Here’s a good example of Natural Language Geocoding. US companies typically pay employees every 2 weeks not every month. What’s good about Snowflake? A few developers who explored it mentioned that: Its ability to scale up compute automatically makes queries run faster. “Time travel” allows you to see how data looked at any point in time and that is impressive and useful. Live data sharing with access control without the need for ETL pipelines is useful. Open-source competition: ClickHouse, Apache Druid, and Presto/Trino DataBricks is a lakehouse and less a data warehouse. It’s more about: storing unstructured data (Snowflake prefers semi-structured: JSON, Avro, etc.) running collaborative notebooks in Python, SQL, Scala, R (Snowflake encourages SQL) I subscribed to ChatGPT Pro mainly for DeepResearch. Here are the first 50 reports I generated: uv Package Manager Overview DuckDB Analytics Comparison Rust vs Python / JavaScript Modern Data Engineering Course LLM Code Migration Practices Cloud Cost Optimization Strategies LLM Coding Interview Tools Report (compare with Perplexity) Text To Speech Engines Customer Service in Indian Public Sector Banks LLMs in Software Development Old version 1: Gen AI in Software Development Old version 2: Gen AI in Software Development Leadership Training Content Open-Source HTTP Servers. Caddy wins. Deep Research Use Cases Nagpur No-Parking Violations Data Science in Food Services Deep Research Disruption to Research Firms LLMs in Design Thinking EU Taxonomy Report Clarification Shell Valuation Analysis Inquiry LLMs in DSLs Research Public API-Based Data Storage Options. Supabase wins. Front-End JS Frameworks Analysis Database Evaluation Guide CSS Frameworks Evaluation Guide CI/CD Tooling Ecosystem Report Color Names Count S Anand Biography. Meh, I know more about me, and it gets a few things wrong. Cosmere Secrets Encyclopedia. This is the best. Deep Research is great if it’s stuff I actually want to read, rather than just learn about. DBT course Future of Coding AI Claude Artifacts Use Cases. This is the only one that managed to get artifacts links correct. I used this for an article for The Hindu. MCP Servers and Clients Research. Learnings: Practically any “tool” can be an MCP server: file systems, APIs, codebases, browsers, collaboration platforms, memory, etc. Most platforms have (or are) integrating MCP. Clients: code editors, chat, and automation tools support MCP. GenAIScript is a good starting point. Tester MCP Client is a browser-based test environment. mcp-cli-client is a CLI-based client mcp-chatbot is a chatbot client Data Moats by Industry Attorney Profile Research Social Media Data APIs Adobe Software Alternatives LLM Hallucination Visualization Techniques API vs Self-hosting Cost Analysis: Always use APIs, avoid self-hosting models. AGI Preparation AGI will emerge step by step. Knowing which step is next will help AI native organisations will emerge in each of these areas. AI design agencies and AI creative Agencies being one example Networking, empathy, leadership have more value now. So will human AI bridging roles (e.g. AI managers, AI consultants, ethics auditors) What’s the value of a human when technology can do everything better? How did this play out in drama (decay) or sports (centralization) or music (globalization)? Modern digital note taking Voice note taking is the game changer Automatically popping of notes based on context such as people places or conversations will be a thing Local LLM Search Tools Blog Post to research paper on copying - suggestions Linux Dev Migration Guide Raspberry Pi SIM options Linux Dev migration guide HTML to JATS conversion LLM context splitting strategies Strategy for AI services in Publishing Gemini multi model editing use cases by industry Pharma Conference Participation Guide I learnt what a Memoji is for the first time. An avatar that follows your facial expressions. Cool! Google shows US flight timings from FlightView. Emperically, based on one data point (my UA-2168 which was delayed by 4 hours), it gets updates faster than Flight Radar 24 or FlightAware or FlightStats. When comparing Indian graduates with their western counterparts, the Indian ones are often seen as: 🟢 Theoretically sound 🟢 Analytical & technical 🟢 Academically disciplined 🟢 Resilient under pressure 🟢 Committed continuous learners 🔴 Rote-learning oriented 🔴 Limited independent inquiry 🔴 Limited creative innovation 🔴 Restricted practical exposure 🔴 Poor communicators 🔴 Low leadership / initiative 🔴 Need structured guidance 🔴 Struggle to network HuggingFace has a “Model tree” against each model that shows the model’s ancestors and descendants. For example, as of now, Deepseek R1 has 75 adapters, 154 finetunes, and 23 quantizations. Perplexity is now powered by Cerebras, which makes their inference as fast as Google. Source. The speed is a big factor, and I’ve switched my default search engine from Google to Perplexity, at least for now. Interview Coder is a desktop app that offers live interview support for coding interviews. It’s a transparent window that reads your screen and answers questions for you. (Given this, I think we need an interviewer support system that tells interviewers what to ask!)