evaluation on S Anand

IIM Bangalore PGP Interview Panel

Tue, 17 Mar 2026 06:42:32 +0530

Yesterday, I was part of an IIM Bangalore interview panel at Hyderabad, along with Professor Subhabrata Das and Debajyoti. Panels typically comprise of two faculty and an alumni, and handle 8 interviews in the morning and eight in the evening, though in our case, we had 9 each.

As we arrived, we were given a USB drive with the student’s resume, statement of purpose, and other documents that they had submitted, which included employment contracts, declarations, letters of recommendation, etc., depending on the student. Each interview was approximately 20 minutes. Luckily, Dr Das set a timer for 18, so we didn’t go too far beyond.

There was practically no time to read the documents before each interview, so I used Claude with the following prompt to suggest questions.

In the context of the below, suggest deep, probing questions that will reveal the suitability of the attached candidate for IIM PGP.

Feel free to search online for additional information about the candidate.

…followed by

The instructions email from IIMB
The attachment in the email
Claude’s earlier advice on how to evaluate candidates based on the above. This included suggestions to:
- Probe for internal consistency: “Your SoP says X but your job history suggests Y. Explain.”
- Probe for authenticity: stuff that coaching can’t teach. (It gave candidate-specific questions.)
- Probe for social sensitivity: “You’re from village X. Where does your aspiration help / hurt whihc segments of your village?”
The candidate’s documents (resume, statement of purpose, etc.)

It helped that there were three panelists. That gave me time to screen the questions and understand the good ones - while others asked their questions.

DISADVANTAGE: You have to spend time reading the questions. Not all questions are great. Hallucinations exist (It said one candidate had a claw-back in their employment contract - they didn’t.)
ADVANTAGE: You pick up little-known stuff. One wrote a monthly salary instead of yearly and didn’t know it. The gaps in the resume, the dips in transcripts, etc. surfaced instantly.

Initially, I just read out some questions.
Then I tried using my own judgement.
Then I went back to Claude’s questions.
It took a while to learn how to use it well.

The interview questions idea came from Claude. It also suggested:

Transcribe real time and pop-up questions. I wish I could and I’m sure it’ll happen soon.
Review scores & transcript with Claude. I did almost exactly that!

I told GitHub Copilot CLI (I had credits) running Claude 4.6 Sonnet (a sensible model):

You are evaluating candidates for the PGP Program at IIM Bangalore.
Read interview-guidelines.md fully to understand the process and criteria.

Then, go through each candidate’s folder. These contain:

notes.md: interview notes

documents.md: documents submitted (converted from the PDF in that folder).

Primarily based on the notes (using documents.md for reference - no need to read PDFs), evaluate all candidates on the criteria mentioned in interview-guidelines.md.

Plan like an expert interview evaluator first. In this context, first think about:

What patterns would an expert in this field check / recognize that beginners would miss?

What questions would an expert ask that a beginner would not know to?

What problems / failures would an expert anticipate that beginners may not be aware of?

How would an expert analyze this? At each step, explain what they are looking for and why.

Document and plan your process in eval-copilot-claude/plan.md.
Then, analyze students (use sub-agents as required) and document your analysis and evaluation in eval-copilot-claude/evaluation.md.

It reviewed all 18 candidates summarizing their background, work-experience, and interview scores along with strengths, concerns, and notable moments, with an evaluation summary.

This serves as a good cross-check, I think. So I told it to:

Compare the ratings against my ratings (and notes) in README.md.
Sort the candidates based on the difference between my ratings vs your (normalized / scaled from 1-10) ratings.
For the biggest differences, analyze the reasons for the differences.
Specifically, what might I have missed in the candidate that I should take a closer look at?
Append this to eval-copilot-claude/evaluation.md.

I ran this on Copilot as well as Claude. We agreed on the best candidate. No question about that. Interesting!

This revealed some interesting observations. I seem to:

Ignore social insensitivity and artistic ability. Most likely because I lack both. Note to self: I’m blind to, and hence devalue, what I’m not good at.
Value genuineness far more than analytics. I rated one candidate with terrible analysis skills as only a marginal reject. Claude rejected them outright. This is true for another candidate, too. But I rejected fakers outright.
I calibrate for preparation. I expect more from those with more experience, with more interview preparation, etc. Not a bad thing - just an observation.

In short:

AI is great for preparation. I should make it easier to feed it context.
AI is great at post-mortems. I need to budget time for this.
AI can help during discussions. I need to figure out the technology for this.

LLMs Are Smarter Than Us

Tue, 01 Jul 2025 06:46:10 +0000

LLMs are smarter than us in many areas. How do we control them?

It’s not a new problem.

VC partners evaluate deep-tech startups.
Science editors review Nobel laureates.
Managers manage specialist teams.
Judges evaluate expert testimony.
Coaches train Olympic athletes.

… and they manage and evaluate “smarter” outputs in many ways:

Verify. Check against an “answer sheet”.
Checklist. Evaluate against pre-defined criteria.
Sampling. Randomly review a subset.
Gating. Accept low-risk work. Evaluate critical ones.
Benchmark. Compare against others.
Red-team. Probe to expose hidden flaws.
Double-blind review. Mask identity to curb bias.
Reproduce. Re-running gives the same output?
Consensus. Ask many. Wisdom of crowds.
Outcome. Did it work in the real world?

For example, you can apply them to:

Vibe coding: Non-programmers might glance at lint checks (Checklist) and see if it works (Outcome).
LLM image designs: Developers might check if a few images look good (Sampling) and check a few marketers (Consensus).
LLM news articles: An journalist might run a Checklist, a Double-blind review with experts, and Verify critical facts (Gating).

You already know many of these. You learnt them in Auditing. Statistics. Law. System controls. Policy analysis. Quality engineering. Clinical epidemiology. Investigative journalism. Design critique.

Worth brushing up on these skills. They’re more important in the AI era.

ChatGPT: https://chatgpt.com/share/6863733f-4ebc-800c-ad3f-a2b472d9e9ca

How To Control Smarter Intelligences

Tue, 01 Jul 2025 06:37:40 +0000

LLMs are smarter than us in many areas. How do we manage them?

This is not a new problem.

VC partners evaluate deep-tech startups.
Science editors review Nobel laureates.
Managers manage specialist teams.
Judges evaluate expert testimony.
Coaches train Olympic athletes.

… and they manage and evaluate “smarter” outputs in many ways:

Verify. Check against an “answer sheet”.
Checklist. Evaluate against pre-defined criteria.
Sampling. Randomly review a subset.
Gating. Accept low-risk work. Evaluate critical ones.
Benchmark. Compare against others.
Red-team. Probe to expose hidden flaws.
Double-blind review. Mask identity to curb bias.
Reproduce. Re-running gives the same output?
Consensus. Aggregate multiple responses. Wisdom of crowds.
Outcome. Did it work in the real world?

For example:

Vibe coding: Non-programmers might glance at lint checks (Checklist) and see if it works (Outcome).
LLM image designs: Developers might check if a few images look good (Sampling) and check a few marketers (Consensus).
LLM news articles: An journalist might run a Checklist, a Double-blind review with experts, and Verify critical facts (Gating).

You already know many of these. You learnt them in Auditing. Statistics. Law. System controls. Policy analysis. Quality engineering. Clinical epidemiology. Investigative journalism. Design critique.

Worth brushing up skills. They’re more important in the AI era.

It's so easy to fool yourself

Sun, 30 Mar 2025 01:45:05 +0000

I asked 4 LLMs to “Suggest catchy rhyming slogans to promote journaling.” Here are the suggestions:

GPT 4.5
- Write it right, day or night!
- Reflect, inspect, journaling perfect!
- Express to impress, journaling’s best!
- Clear your mind, write and unwind!
- Pen your zen, again and again!
- Daily pages, inner sages!
- Take a pause, write your cause!
- Emotions flow, journals grow!
- Write, ignite, your inner light!
- Ink it, think it, don’t just blink it!
- Pen your quest, ease your stress!
- From confusion to conclusion—journal your solution!
- Keep a book, take a look, watch your growth hook!
- Thoughts on paper, worries taper!
- Write to heal, make it real!
Claude 3.7 Sonnet
- Write it down, turn that frown upside down!
- Pen your thoughts, connect the dots.
- Journal every day, keep the stress away.
- Ink your mind, treasures you’ll find.
- Write and reflect, your future self will thank you direct.
- Scribble and doodle, your mind will stay in the noodle.
- Journal with flair, show yourself you care.
- Jot it in ink, give yourself time to think.
- Fill the page, release the rage.
- Write your story, capture the glory.
- Daily writing, life enlightening.
- Express in a line, your day will shine.
- Thoughts on paper, life feels safer.
- Write with heart, that’s the smart start.
- Journaling’s neat, makes life complete.
Gemini 2.5 Pro
- Thoughts unclear? Write them down here.
- Let the ink help you think.
- Untangle your mind, one line at a time.
- To make things clear, keep your journal near.
- When your head’s in a knot, give journaling a shot.
- Feeling stressed? Put your pen to the test.
- Write it down, let worries drown.
- Ease your mind, leave troubles behind.
- Let your feelings take flight, just sit down and write.
- Turn the page on inner rage.
- Write to know, watch yourself grow.
- Explore your soul, take control.
- Pen and page, turn a new age.
- Discover your way, write every day.
- Know your heart? Make journaling your art.
- Capture the day in your own way.
- Make moments last, write down the past.
- Don’t let memories stray, write them today.
- Reflect and connect, gain self-respect.
- Life moves fast? Make memories last.
- Get in the flow, write and grow.
- Pen in hand, understand.
- Daily lines, clear designs.
- Just take note, stay afloat.
- Write your heart, make a fresh start.
Grok 3:
- Write your day, let thoughts play!
- Pen in hand, your mind expands!
- Journal your way, seize the day!
- Feel the groove, let words move!
- Scribe your soul, make it whole!
- Ink flows free, just be me!
- Page by page, unlock your sage!
- Write it down, wear the crown!

At first glance, GPT 4.5 didn’t impress me. Claude 3.7 Sonnet did. I also didn’t like Gemini 2.5 Pro, but Grok was great.

Grok 3 > Claude 3.7 Sonnet > Gemini 2.5 Pro > GPT 4.5.

But it’s hard to compare a dozen quotes at once. So I made a small quotes arena app to help me pick my favorites. It shows me random pairs of quotes and asks which I like more.

To my surprise, after answering 30+ “games” in the arena, I found that based on my preferences:

Claude 3.7 Sonnet > Gemini 2.5 Pro > GPT 4.5 > Grok 3.

That was weird. I thought I liked Grok’s results a lot. I continued till I answered 50+ games. Then I found that:

Grok 3 > GPT 4.5 > Gemini 2.5 Pro > Claude 3.7 Sonnet.

That’s the exact opposite of the previous result.

Honestly, I’m depressed. I’ve learnt 3 things:

I can’t judge stuff at a glance.
But I think I can (especially with code.)
Even when evaluating carefully, my preferences are unstable.

Nothing has shaken my confidence more in recent times. I cannot trust my judgement. I need written evals. Badly.

LLMs still do not locate bounding boxes well

Fri, 01 Nov 2024 16:20:03 +0000

I sent an image to over a dozen LLMs that support vision, asking them:

Detect objects in this 1280x720 px image and return their color and bounding boxes in pixels. Respond as a JSON object: {[label]: [color, x1, y1, x2, y2], …}

None of the models did a good-enough job. It looks like we have some time to go before LLMs become good at bounding boxes.

I've given them a subjective rating on a 1-5 scale below.

Model	Positions	Sizes
gemini-1.5-flash-001	🟢🟢🟢🔴🔴	🟢🟢🟢🟢🔴
gemini-1.5-flash-8b	🟢🟢🟢🔴🔴	🟢🟢🟢🔴🔴
gemini-1.5-flash-002	🟢🟢🔴🔴🔴	🟢🟢🟢🔴🔴
gemini-1.5-pro-002	🟢🟢🟢🔴🔴	🟢🟢🟢🟢🔴
gpt-4o-mini	🟢🔴🔴🔴🔴	🟢🟢🔴🔴🔴
gpt-4o	🟢🟢🟢🟢🔴	🟢🟢🟢🟢🔴
chatgpt-4o-latest	🟢🟢🟢🟢🔴	🟢🟢🟢🟢🔴
claude-3-haiku-20240307	🟢🔴🔴🔴🔴	🟢🟢🔴🔴🔴
claude-3=5-sonnet-20241022	🟢🟢🟢🔴🔴	🟢🟢🟢🔴🔴
llama-3.2-11b-vision-preview	🔴🔴🔴🔴🔴	🔴🔴🔴🔴🔴
llama-3.2-90b-vision-preview	🟢🟢🟢🔴🔴	🟢🟢🟢🔴🔴
qwen-2-vl-72b-instruct	🟢🟢🟢🔴🔴	🟢🟢🔴🔴🔴
pixtral-12b	🟢🟢🔴🔴🔴	🟢🟢🟢🔴🔴

I used an app I built for this.

Here is the original image along with the individual results.

Update

Adding gridlines with labeled axes helps the LLMs. (Thanks @Bijan Mishra.) Here are a few examples:

How does Gemini process videos?

Thu, 24 Oct 2024 08:47:21 +0000

The Gemini documentation is clear:

The File API service extracts image frames from videos at 1 frame per second (FPS) and audio at 1Kbps, single channel, adding timestamps every second. These rates are subject to change in the future for improvements in inference.

Note: The details of fast action sequences may be lost at the 1 FPS frame sampling rate. Consider slowing down high-speed clips for improved inference quality.

Individual frames are 258 tokens, and audio is 32 tokens per second. With metadata, each second of video becomes ~300 tokens, which means a 1M context window can fit slightly less than an hour of video.

To ask questions about time-stamped locations, use the format MM:SS, where the first two digits represent minutes and the last two digits represent seconds.

But on this ThursdAI episode: Oct 17 - Robots, Rockets, and Multi Modal Mania…, at 1:00:50, Hrishi says

I don’t think it’s a series of images anymore because when I talk to the model and try to get some concept of what it’s perceiving, it’s no longer a series of images.

If that’s the case, it’s a huge change. So I tested it with this video.

This video has 20 numbers refreshing at 4 frames per second.

When I upload it to AI Studio, it takes 1,316 tokens. This is close enough to 258 tokens per image (no audio). So I’m partly convinced that Gemini still processing videos at 1 frame per second.

Then, I asked it to Extract all numbers in the video using Gemini 1.5 Flash 002 as well as Gemini 1.5 Flash 8b. In both cases, the results were: 2018, 85, 47, 37, 38.

These are frames 2, 6, 10, 14, 18 (out of 20). So, clearly Gemini is still sampling at about 1 frame per second, starting somewhere between 0.25 or 0.5 seconds.