research-workflows on S Anand

How to use AI for research

Sat, 04 Apr 2026 23:30:53 +0800

I asked ChatGPT to research universities’ AI policies. Here is the report

Here are the four lessons I learned from that - about how to use AI for research.

1. Show examples of failures to avoid. Jivraj’s earlier research kept surfacing AI policies universities had researched, not written for themselves!. So I told ChatGPT to:

… double-check that they ARE, in fact, about their own use of AI - not policies they’re proposing for others or are researching.

This is called pre-specifying exclusions. Giving negative examples help. Wei (2022).

2a. “Double-check” doesn’t always work. Though I told ChatGPT to “double-check”, it still got things wrong. For example, it cited MIT’s AI policy home page as evidence that the policy covers students and faculty, just because the words were present. That’s not right!

Models get over-confident - and that’s exactly when they don’t double-check. Asking them to double-check is a good habit, but not fail-safe. Kadavath (2022)

2b. Expicitly tell it to find mistakes. I told it to:

Find mistakes in as many claims as you can.

This is stronger than “double-check”. It turns the model against itself, and it worked quite well.

1. Show examples of failures to avoid. (Repeat.) When asking it to find mistakes, I gave it the same example.

… MIT, “covers_faculty_or_staff” cites “quote”: “Students • Faculty and Staff • Visitors and Guests • Generative AI use at MIT”. But that’s actually a set of links to Students, Faculty and Staff, etc. It’s not evidence that the POLICY covers them - and I’m quite sure the policy isn’t for guests!

That’s few-shot prompting. Concrete examples beat abstract instructions.

3. Ask it to list failures explicitly. I told it:

I am also interested in universities that conspicuously lack a policy …

Without that, it might have returned only positive hits. Missing evidence and failures are important data, too!

4. Break large tasks into batches. When I asked it to research 20 universities, it made several mistakes. Instead:

This may be a complex task, so let’s just do this for the first four Universities.

Now, it didn’t make any mistakes! Sometimes, it gets lost in the middle for long tasks.

So there it is - the four rules of AI research I learned from this exercise:

Show examples of failures to avoid
“Double-check” doesn’t always work. Expicitly tell it to find mistakes
Ask it to list failures explicitly.
Break large tasks into batches.

O3 Is Now My Personalized Learning Coach

Sun, 20 Apr 2025 12:10:53 +0000

I use Deep Research to explore topics. For example:

Text To Speech Engines. Tortoise TTS leads the open source TTS.
Open-Source HTTP Servers. Caddy wins.
Public API-Based Data Storage Options. Supabase wins.
etc.

But these reports are very long. With O3 and O4 Mini supporting thinking with search, we can do quick research, instead of deep research. One minute, not ten. One page, not ten.

For example, I asked O3 to automate my weekly tech learning process:

From the links at https://www.hntoplinks.com/week pick those that would be of interest to me.
Read the articles and explain the key insights – what’s non-obvious and how can I use it.
When sharing how I can use it, research alternate approaches and suggest which approach is best.

This brings two new capabilties I love.

Memory. I said, “… that would be of interest to me”. Given my past chats and context, it really knows me. For example, the reasoning trace shows:

I’m narrowing down the list for Anand, focusing on his interests like generative AI, open-source tools, and minimal libraries.

That’s exactly right!

Long reasoning. It reads each site step by step, unsupervised. Here’s a brief reasoning trace:

Searched the web
Got the Unsure Calculator article. Now, onto the next one: the Hackable AI assistant with SQLite cron jobs. I’ll open it and dive into the details. I think this will be a great fit for Anand, considering the focus on AI tools and development. It should provide some interesting insights!
Searched the web
Now that we’ve moved through a couple of articles, it’s time to tackle the Gemini 2.5 Flash AI model. I think this one will be particularly interesting for exploring advancements in AI. Let me grab it and dive into the details so I can distill the insights into something relevant.
Searched the web
Next up: I’ll open the OpenAI blog article on GPT-4.1 in API. This could bring in some cool technical
…

The result is fantastic. While on a trouser shopping trip, I learnt more than when I browse Hacker News:

“Unsure” is a tool to calculate uncertainty formulas, like the uncertainties Python package.
Responses API adds features like linkable history, mid‑chat truncation, flexible reasoning detail, and more.
Augment Code is an AI code editor that’s growing popular on Reddit.
GPT 4.1’s 75% discounted prompt caching (instead of 50%) gives them an edge on repetitive tasks. OpenAI
Nix flakes are a reliable alternative to DevContainers that don’t need Docker - but don’t work on Windows.
TLS certificates will expire in 47 days from 15 Mar 2020. Automated domain renewals are a must. Digicert
… and a bunch of other things.

Here are Hacker News summaries for a month or specific days:

Mar 2025: Mistral OCR is a dedicated OCR, not an LLM. DuckDB UI has notebooks!
13 Apr 2025: Learnt about Signalbloom which automates Q1 summaries and this MCP Primer
15 Apr 2025: 12-factor agents shares agent lessons. Send JSX instead of JSON. AI as Normal Technology.

This enables:

Personalized learning, i.e. it tells me what I don’t know, and how I can apply it. This is powerful.
Learning on the go, e.g. via voice while cycling or walking.
Learning from untapped sources. This includes: GitHub repos, research papers, open data registries, patent filings, earnings transcripts, subreddits, judgements or acts, open data repositories or collections, and many more.

Take GitHub as an untapped source. I asked O3:

Go through the OpenAI Codex CLI repo on GitHub.
Teach me innovative, new, and surprising techniques or approaches or libraries I might not know about.
For each, explain what is interesting, how I might use it, and how this approach contrasts with alternatives.

This was a treasure trove, too! I learn about:

Ink: React for CLI!
Zod: schema validation
Creating firewalls
Fuzzy path matching
… and a bunch of other things

I tried O4 Mini High and saw similar results. I felt O3 still gave me more personalized suggestions.

Let’s see what we can learn from pull requests on Codex.

Go through all the pull requests that have been merged into OpenAI’s codecs repository on GitHub. Pick the ones that would be most interesting to me. You can group a few if they are very related and give me the top 10 most interesting PRs that would be relevant for me. Also explain why these are relevant to me, how I might use them, and any interesting details about the way in which the PRs were written.

Soon, we won’t just follow a lesson plan – we’ll have lessons built just for us. AI will track how we learn and adapt in real time. It’ll feel like having a personal coach in your back pocket. That future starts now.

This also opens a door to endless curiosity. There’s no limit to what we can explore. Curiosity is the competitive advantage, now.

Things I Learned - 16 Feb 2025

Sun, 16 Feb 2025 00:00:00 +0000

This week, I learned:

Connected Papers shows papers similar to each other based on co-citation and bibliographic coupling for ~50,000 papers.
Notes from a fireside chat with Prashanth Chandrasekar, CEO, StackOverflow, and the StackOverflow team
- There’s a signal that software demand is growing in 2024. Many more students took the StackOverflow survey in 2024. So more students (or other professionals) are shifting into / starting to learn software development.
- The AI Index is a good resource for AI trends.
- Experts are better able to use AI for writing code. Less experienced developers are more likely to use AI for code reviews, project planning, etc.
- There’s a 5% decline in favorability for AI tools compared to 2023, maybe due to disappointing results.
- Pilot groups working on AI are 25-30% more productive. They’re the most enthusiastic. For the rest of the company, it drops off to 5-10%
  - #LEARNING Benefit comes from NEW people becoming programmers, not existing ones getting more effective?
- StackOverflow wants to be where the developer is.
  - The programmer workflow was: Google -> StackOverflow -> GitHub.
  - Now it’s changing to ChatGPT / Cursor -> GitHub.
  - StackOverflow has a partnership with OpenAI and working on a plugin. Same with Google’s Duet AI, GitHub Copilot, many others. They’ll link to StackOverflow.
- StackOverflow is driving integration actively through an enterprise Overflow API
- Q: What tech have you seen blaze through the ranks?
  - Prashanth: Abstraction wins. Stuff that abstracts away things well and more wins. This includes Gen AI.
  - Erin Yepis: Rust (from 3% to 12%). AWS has steady growth.
  - Erin Yapis: I have a time series spreadsheet that I’ll publish.
- Q: What technologies are unusually tightly coupled?
  - Prashanth: AWS & Google Cloud are tightly coupled.
- Q: We have an engagement problem. Might be India-specific. What are low-effort high-return mechanisms to increase engagement.
  - Eric Woodring: Rather than a static web page, integrate it using the API. #TODO
  - Ben Marconi: Use LLMs to write post mortems and push to StackOverflow. #TODO
  - Eric Woodring: “Hydrating” the community helps.
    - We take repeat questions on Teams / Slack and seed them using LLMs.
    - We integrate with the API to auto-add Q&A.
    - Transform documentation into Q&A. Potentially UPDATE existing Q&A if it’s wrong.
- Q: What unexpected lessons about developer behavior have you learned while running StackOverflow?
  - Prashanth: We didn’t expect developers moving away from Google. Now it moved to the IDE.
- Q: What are you learning about developer learning behavior?
  - Ben Marconi: Generating LLM-based onboarding documents.
  - Using StackOverflow for Teams to identify who the experts are to contact for specific topics.
- Q: Are you thinking about leveraging Stack Overflow’s knowledge base for personalized or interactive learning experiences? How?
  - Prashanth: Traditionally, people use StackOveflow for productivity, learning, and flexibility (i.e. to ask/answer questions asynchronously without breaking their flow). So yeah, learning is important for us. (Duh!)
- Q: Could Stack Overflow’s interactions help evaluate the accuracy and relevance of LLM-generated code? Or provide potential metrics on quality?
  - Prashanth: LLM accuracy improves by ~30%. Upvotes / downvotes are reinforcement learning (RL) in steroids, so that helps.
- Q: What are your thoughts on reliance on LLMs potentially deskill-ing developers?
  - Prashanth: A real issue for junior developers, not for senior ones.
  - They’ll come across as knowledgeable.
  - Make internal evaluations and interviews more rigorous.
- Anand’s requests for action:
  - Could I get a copy of Erin’s spreadsheet? Vivek Narayanan will follow-up.
  - Could you help me learn more about hydration? Nick Madison will set up a meeting with customer success group.
I switched to fish shell mainly because:
- Autocomplete and tab completion works perfectly, out-of-box.
- Syntax highlighting is beautiful
- Great multi-line editing
To format with VS Code Ruff, you need to point the ruff.interpreter setting to a Python interpreter. You can’t run the ruff server without Python, even though ruff itself doesn’t need Python.
cd checks all paths specified in CDPATH for the directory name and changes to the first match. That’s pretty convenient!
Flipper Zero is now on my list of “To Buy” tools. It has a variety of hardware devices including NFC, RFID, Bluetooth, Infrared, etc. and is great to reverse engineer or hack devices.