llm-comparison on S Anand

Sonnet 4.6 vs MiniMax M2.7

Tue, 24 Mar 2026 17:06:02 +0800

Based on several (i.e. two) recommendations, I subscribed to MiniMax. At $10/month, you get 1,500 requests every 5 hours and 15,000 every week. That’s a LOT!

Using the same prompt I had Claude Code generate two data stories:

The first paragraph, by Claude Sonnet 4.6

The first paragraph, by MiniMax M2.7

Here’s my comparison of the two. It’s partly based on Claude Opus 4.6’s comparison but I felt the same way.

Dimension	Sonnet 4.6	MiniMax M2.7
Narrative quality	Immersive
Content coverage	Comprehensive
Visual design	More varied, ambitious bands, no errors
CSS		Better use of CSS variables
Tooltips	Richer, comprehensive, `data-tip`
Modals/popups	Richer, more types, more details
Animated SVGs	Richer, visually distinctive, sophisticated
Slides	Larger readable grid
Code samples	XML vs JSON-LD side-by-side
External references	Far more authoritative links
Accessibility	ARIA, keyboard, alt text
Generation quality	Clean, no Chinese character artifacts

In other words, Sonnet 4.6 is a clear winner on nearly every dimension.

But the cost factor is too big a difference to ignore. It feels like a 10x difference. So the question probably is: what can I do with a reasonably good model that can generate 10X the quantity at the same price?

(To be fair, GPT 5.4 Mini at 75c/MTok and Gemini 3 Flash at 50c/MTok are not far from MiniMax M2.7 at 30c/MTok - but their code quality seems lower. I generated a Codex - GPT 5.4 Mini version and while it has fewer errors it has even less visual style and narrative quality.)

Computer use feels like a candidate. I used Rodney to research what drives my LinkedIn reach & engagement, and update my SKILL.md.

I could try experimenting with sub-agents, doing bulk analysis (e.g. of code, transcripts, images), data discovery, etc. The crux of these is parallelization - something I have not explored much.

It looks like twe’re entering an era where there are two kinds of use cases: high-quality for the best models, large-scale for the cheap models. The question is: how do I make the most of both?

Source Code

UPDATE: Cheap models (or at least MiniMax M2.7) may be far less useful than I thought. I used MiniMax M2.7 with Claude Code for:

24 Mar 2026: Email analysis. I had it review my 15-year Gramener email archive for key events for a book. But it fetched too few results, so I switched to Codex (GPT 5.4 xhigh).
25 Mar 2026: Capture The Flag. But it couldn’t solve problems, so I switched to Codex (GPT 5.4 xhigh).
25 Mar 2026: Songs download. I had it find popular Tamil songs and download them from YouTube. But the metadata was poor, so I switched to my own song collection.
26 Mar 2026: LEAN proofs. It started making too many basic mistakes (spelling errors in code!) I switched to Copilot (GPT 5.4 xhigh).
29 Mar 2026: Calvin & Hobbes image analysis. It couldn’t even read the images and confidently saw “Hobbes stuck to a baseball bat with Mom & Dad” in a strip that only featured Calvin & Susie.

The main problems are:

It errs confidently. It doesn’t do ROT13 well. It can’t see images. It mis-understands error messages. It assigned my earlier company’s incorporation date (NGIMAGE) as Gramener’s. It made Vijay Sethupathi a lyricist. When a process failed with just 12% coverage, it just continued. It just reported what’s done, not what’s missing.
It’s a slow learner. For picoCTF, it had the pieces but couldn’t assemble them. Claude Code resets the cwd, but it never switched to absolute paths. It mixed uv run with python3. It rewrites, resets or waits instead of diagnosing.

It’s best for simple, single-step tasks. Not where knowledge, accuracy, research matters. When using it, keep tasks small and verify correctness, completeness.

Which LLMs get you better grades?

Fri, 06 Mar 2026 19:26:47 +0800

In my graded assignments students can pick an AI and “Ask AI” any question at the click of a button. It defaults to Google AI Mode, but other models are available. I know who uses which model and their scores in each assignment.

I asked Codex to test the hypothesis whether using a specific model helps students perform better.

The short answer? Yes. Model choice matters a lot. Across 333 students, here’s how much more/less students score compared with ChatGPT:

Perplexity: increases scores by ~11% points (99.8% sure it has an impact).
Claude: +9% (99.5% sure)
Clipboard:, i.e. just copying to the clipboard and pasting in their choice of AI: +9% points (97.5% sure)
Google: -1% (Not sure. Sorry, Sundar.)

If you are still using default ChatGPT, you are leaving nearly a full letter grade on the table compared to Claude or Perplexity users.

I also looked at exam timing. Do specific models perform better under last-minute pressure?

Statistically, no. The model-by-timing interaction isn’t significant (92% sure). A good model won’t save a rushed submission any better than a bad one.

But procrastination does hurt, regardless of which AI they frantically prompt. In the first graded assignment (GA1), there was a negative correlation (-37%, 100% sure) between submission time and final score.

BTW, I correlated with the most-used model. I can’t guarantee they used that specific model on every single attempt.

Of course, correlation isn’t causation. Maybe Claude writes better code. Or perhaps the kind of student who consciously switches from ChatGPT to Perplexity or Claude is just a better, more engaged student.

But a 9-to-10 point bump is a pretty good reason to experiment.

It's so easy to fool yourself

Sun, 30 Mar 2025 01:45:05 +0000

I asked 4 LLMs to “Suggest catchy rhyming slogans to promote journaling.” Here are the suggestions:

GPT 4.5
- Write it right, day or night!
- Reflect, inspect, journaling perfect!
- Express to impress, journaling’s best!
- Clear your mind, write and unwind!
- Pen your zen, again and again!
- Daily pages, inner sages!
- Take a pause, write your cause!
- Emotions flow, journals grow!
- Write, ignite, your inner light!
- Ink it, think it, don’t just blink it!
- Pen your quest, ease your stress!
- From confusion to conclusion—journal your solution!
- Keep a book, take a look, watch your growth hook!
- Thoughts on paper, worries taper!
- Write to heal, make it real!
Claude 3.7 Sonnet
- Write it down, turn that frown upside down!
- Pen your thoughts, connect the dots.
- Journal every day, keep the stress away.
- Ink your mind, treasures you’ll find.
- Write and reflect, your future self will thank you direct.
- Scribble and doodle, your mind will stay in the noodle.
- Journal with flair, show yourself you care.
- Jot it in ink, give yourself time to think.
- Fill the page, release the rage.
- Write your story, capture the glory.
- Daily writing, life enlightening.
- Express in a line, your day will shine.
- Thoughts on paper, life feels safer.
- Write with heart, that’s the smart start.
- Journaling’s neat, makes life complete.
Gemini 2.5 Pro
- Thoughts unclear? Write them down here.
- Let the ink help you think.
- Untangle your mind, one line at a time.
- To make things clear, keep your journal near.
- When your head’s in a knot, give journaling a shot.
- Feeling stressed? Put your pen to the test.
- Write it down, let worries drown.
- Ease your mind, leave troubles behind.
- Let your feelings take flight, just sit down and write.
- Turn the page on inner rage.
- Write to know, watch yourself grow.
- Explore your soul, take control.
- Pen and page, turn a new age.
- Discover your way, write every day.
- Know your heart? Make journaling your art.
- Capture the day in your own way.
- Make moments last, write down the past.
- Don’t let memories stray, write them today.
- Reflect and connect, gain self-respect.
- Life moves fast? Make memories last.
- Get in the flow, write and grow.
- Pen in hand, understand.
- Daily lines, clear designs.
- Just take note, stay afloat.
- Write your heart, make a fresh start.
Grok 3:
- Write your day, let thoughts play!
- Pen in hand, your mind expands!
- Journal your way, seize the day!
- Feel the groove, let words move!
- Scribe your soul, make it whole!
- Ink flows free, just be me!
- Page by page, unlock your sage!
- Write it down, wear the crown!

At first glance, GPT 4.5 didn’t impress me. Claude 3.7 Sonnet did. I also didn’t like Gemini 2.5 Pro, but Grok was great.

Grok 3 > Claude 3.7 Sonnet > Gemini 2.5 Pro > GPT 4.5.

But it’s hard to compare a dozen quotes at once. So I made a small quotes arena app to help me pick my favorites. It shows me random pairs of quotes and asks which I like more.

To my surprise, after answering 30+ “games” in the arena, I found that based on my preferences:

Claude 3.7 Sonnet > Gemini 2.5 Pro > GPT 4.5 > Grok 3.

That was weird. I thought I liked Grok’s results a lot. I continued till I answered 50+ games. Then I found that:

Grok 3 > GPT 4.5 > Gemini 2.5 Pro > Claude 3.7 Sonnet.

That’s the exact opposite of the previous result.

Honestly, I’m depressed. I’ve learnt 3 things:

I can’t judge stuff at a glance.
But I think I can (especially with code.)
Even when evaluating carefully, my preferences are unstable.

Nothing has shaken my confidence more in recent times. I cannot trust my judgement. I need written evals. Badly.