model-comparison on S Anand

Sat, 25 Oct 2025 00:00:00 +0000

I asked multiple coding agents and models to build the same app:

Create a single-page web app at index.html that beautifully renders a GitHub user profile and activity comprehensively. Pick the ID in the URL ?id=…, default to ?id=torvalds.

… and compared their quality, cost, and speed.

My observations:

Quality variance is the highest. Some models / agents produce great visuals, some average, some fail completely.

Cost and time variance are lower among the successful models. About 2X variance in each.

This is unlike non-code usage, where quality varies less than cost.

My takeaway: Pick the best model / agent. Don’t worry about speed and cost - the variance is lower.

Results: https://sanand0.github.io/llmevals/coding-agents/

Thu, 05 Jun 2025 05:36:05 +0000

I tried out GPT Image 1.5. It adds more contrast, ink, texture, detail, and polish. See https://sanand0.github.io/llmartstyle/?category=pop

It’s more powerful when generating different infographic styles: https://sanand0.github.io/llmartstyle/?category=text

But it’s still terrible at faces. Overall, better competition for Nano Banana. Not yet dethroning Nano Banana Pro for me.

Fri, 18 Apr 2025 13:10:15 +0000

With the Gemini 2.5 Flash release, Google envelopes the entire cost-quality frontier of LLMs. In other words, at any cost or quality level, today, the best model to use according to the LM Arena score is a Gemini model.

Results for O3, O4 Mini, and GPT 4.1 are not yet on LM Arena. But until then, #Google dominates. Nice work!

Link: https://sanand0.github.io/llmpricing/

ImageGen 3 is the top image model now

Mon, 10 Mar 2025 04:03:42 +0000

Gemini’s ImageGen 3 is rapidly evolving into a very powerful image editing model. In my opinion, it’s the best mainstream image generation model.

Ever since it was released, it’s been the most realistic model I’ve used. I’ve been using it to imagine characters and scenes from The Way of Kings. For example, when I wanted to visualize Helaran’s first appearance, I just quoted the description:

Draw this. Galloping directly toward them was a massive black horse bearing a rider in gleaming armor that seemed to radiate light. That armor was seamless—no chain underneath, just smaller plates, incredibly intricate. The figure wore an unornamented full helm, and the plate was gilded. He carried a massive sword in one hand, fully as long as a man was tall. It wasn’t a simple, straight sword—it was curved, and the side that wasn’t sharp was ridged, like flowing waves. Etchings covered its length. It was beautiful. Like a work of art. Cenn had never seen a Shardbearer, but he knew immediately what this was. How could he ever have mistaken a simple armored lighteyes for one of these majestic creatures?

And it generated this:

Or, to visualize Highprince Roion:

Draw this. Highprince Roion stood in front of one of the maps, hands clasped behind his back, his numerous attendants clogging the other side of the gallery. Roion was a tall, light-skinned man with a dark, well-trimmed beard. He was thinning on top. Like most of the others, he wore a short, open-fronted jacket, exposing the shirt underneath. Its red fabric poked out above the jacket’s collar.

Here’s the first appearance of Kaladin:

Draw this. A man walked up through the ranks, carrying a shortspear that had two leather knife sheaths strapped to the haft. The newcomer was a young man—perhaps four years older than Cenn’s fifteen—but he was taller by several fingers than even Dallet. He wore the common leathers of a spearman, but under them was a pair of dark trousers. That wasn’t supposed to be allowed. His black Alethi hair was shoulder-length and wavy, his eyes a dark brown. He also had knots of white cord on the shoulders of his jerkin, marking him as a squadleader.

The images are stunning in quality, reproduce the prompt quite faithfully, and cost 2-4 cents each (via the API.)

But this is just the begining of power of ImageGen 3. For those with access, you can edit existing images, e.g.

Inpainting: add stuff to images
Outpainting: extend an image, zooming out
Product editing: replace the background with something else (actually, this is powerful for lots of things beyond products)
Styling: change the style of an image (e.g. photo to comic)
Changing the context: draw this dog but in a different setting
… and more

A fair bit of this can be done by selecting a region (masking) or mask-free.

I’m really looking forward to this.

Sun, 06 Oct 2024 05:48:57 +0000

Which is the most neurotic / emotional #LLM?

I ran the Big 5 personality test on a bunch of LLMs (for my TEDx MDI Gurgaon talk in August.) Here are the results.

https://sanand0.github.io/llmpersonality/

Claude 3 Haiku and Llama 3 8b consider themselves the most emotional models. In fact, some of Llama 3 8b’s quotes are hilarious:

Get stressed out easily. - 4. Moderately Accurate (I can get stressed, but I’m working on managing my stress levels)

Shirk my duties. - 2. Moderately Inaccurate (I try to be responsible and complete my tasks, but I’m not perfect and sometimes procrastinate)

Maybe LLMs don’t fill out personality tests accurately. Humans don’t either. Maybe we should assess behavior, not surveys. But, we use these on humans, so why not LLMs?

Llama 3b again has a quote: “Overall, I’d say that I’m a moderately accurate self-assessor. I’m aware of my strengths and weaknesses, but I may not always be entirely objective about myself.

So, to the extent we believe they’re good self-assessors:

Llama 3.1 8b is a neurotic and disorganized model
Llama 3.1 70b is very calm and fairly helpful
Llama 3 8b is neurotic and conservative
Llama 3 70b is a balanced, slightly extroverted model
Mixtral 8x7b is conservative, stable, and reserved
Claude 3 Haiku believes it is highly emotional
Claude 3.5 Sonnet is reliable, talkative, and stable
GPT-3.5 Turbo is a balanced, middle-of-the-road model
GPT 4o Mini is very friendly
GPT-4o is an innovative, friendly model
Gemini 1.5 Flash is innovative, quiet, opinionated
Gemini 1.5 Pro is mostly open, slightly emotional

Read on for details.

Data & visualization: https://sanand0.github.io/llmpersonality/
Talk recording video: https://youtu.be/UniihAkwvLs
Code: https://github.com/sanand0/llmpersonality
Transcript & slides: https://github.com/sanand0/llmpersonality/tree/main/tedx