<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>computer-vision on S Anand</title>
    <link>https://www.s-anand.net/blog/tag/computer-vision/</link>
    <description>Recent content in computer-vision on S Anand</description>
    <generator>Hugo -- 0.164.0</generator>
    <language>en-us</language>
    <lastBuildDate>Thu, 22 Jan 2026 09:33:02 +0530</lastBuildDate>
    <atom:link href="https://www.s-anand.net/blog/tag/computer-vision/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Gemini copies images almost perfectly</title>
      <link>https://www.s-anand.net/blog/gemini-copies-images-almost-perfectly/</link>
      <pubDate>Thu, 22 Jan 2026 09:33:02 +0530</pubDate>
      <guid>https://www.s-anand.net/blog/gemini-copies-images-almost-perfectly/</guid>
      <description>&lt;p&gt;&lt;strong&gt;Summary&lt;/strong&gt;: Nano Banana Pro is &lt;em&gt;much&lt;/em&gt; better than recent models at copying images without errors. That lets us do a few useful things, like:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Pre-process images for OCR, improving text recognition by cleaning up artifacts while preserving text shapes exactly.&lt;/li&gt;
&lt;li&gt;Convert textbook raster diagrams into clean vector-like images that vectorizers can process easily.&lt;/li&gt;
&lt;li&gt;Create in-betweens for cartoon animations&lt;/li&gt;
&lt;li&gt;Copy torn, stained 1950s survey maps into pristine, high-contrast replicas with boundary lines preserved pixel-perfectly.&lt;/li&gt;
&lt;li&gt;Redraw sewage map blueprints or refinery blueprints into clean schematics, separating the &amp;ldquo;pipes&amp;rdquo; from the &amp;ldquo;background noise&amp;rdquo;.&lt;/li&gt;
&lt;li&gt;&amp;hellip; and more!&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;GPT Image 1.5 has a good reputation for drawing exactly what you tell it to.&lt;/p&gt;
&lt;p&gt;But right now, nothing beats Gemini 3 Pro Image Preview (Nano Banana Pro) on copying images &lt;strong&gt;exactly&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;To check this, &lt;a href=&#34;https://github.com/Nitin399-maker&#34;&gt;Nitin&lt;/a&gt; asked a bunch of models to:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-markdown&#34; data-lang=&#34;markdown&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;Recreate the provided image EXACTLY:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;-&lt;/span&gt; Same resolution and aspect ratio
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;-&lt;/span&gt; Same colors, lighting, composition
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;-&lt;/span&gt; Same shapes and positions
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;-&lt;/span&gt; No stylization, no creativity, no cleanup
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;-&lt;/span&gt; Do not add/remove text, objects, or artifacts
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;-&lt;/span&gt; Output should be the closest possible pixel-level match
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&amp;hellip; and passed it different kinds of images.&lt;/p&gt;
&lt;p&gt;Here&amp;rsquo;s their &lt;a href=&#34;https://en.wikipedia.org/wiki/Structural_similarity&#34;&gt;structural similarity index (SSIM)&lt;/a&gt;:&lt;/p&gt;
&lt;!--

Generated by: https://gemini.google.com/app/624b7e8763d05222

Convert this Markdown table into a HTML table colored using D3 RdYlGn scale (highest value is green, lowest is red), ensuring contrast - i.e. white foreground text if the background is dark and black foreground text otherwise, with values shown as percentages to 1 decimal (e.g. 99.9%), number columns right aligned and the first text column left aligned.

Add a button that lets me copy the rendered HTML to the clipboard keeping the entire formatting (horizontal alignment, foreground and background, no fonts / padding etc.) as inlined styles in the HTML. I want to paste this into another (static) HTML page and get the SAME output.

| Image Type       | gemini-3-pro-image-preview | gemini-2.5-flash-image | gpt-image-1.5 | chatgpt-image-latest | gpt-image-1 | gpt-image-1-mini |
| ---------------- | -------------------------- | ---------------------- | ------------- | -------------------- | ----------- | ---------------- |
| Human portrait   | 0.9986                     | 0.8471                 | 0.9342        | 0.8976               | 0.8948      | 0.7695           |
| Human photo      | 0.9931                     | 0.7609                 | 0.8289        | 0.8215               | 0.7400      | 0.7181           |
| Landscape        | 0.9984                     | 0.9574                 | 0.9309        | 0.9244               | 0.9027      | 0.9203           |
| Cartoon          | 0.9984                     | 0.8126                 | 0.7232        | 0.7185               | 0.7236      | 0.7227           |
| Diagram          | 0.9790                     | 0.3222                 | 0.3235        | 0.2440               | 0.2692      | 0.3164           |
| Scatterplot      | 0.8913                     | 0.8587                 | 0.4627        | 0.4195               | 0.3598      | 0.3845           |
| Surreal painting | 0.9973                     | 0.6669                 | 0.7720        | 0.7656               | 0.7221      | 0.5136           |
| Stereogram       | 0.9850                     | 0.3784                 | 0.4115        | 0.3412               | 0.3614      | 0.3960           |
| Document         | 0.8942                     | 0.3357                 | 0.1397        | 0.1004               | 0.0788      | 0.0832           |
| Tuvalu map       | 0.9620                     | 0.9992                 | 0.7242        | 0.6179               | 0.6447      | 0.6544           |
| Overall Average  | 0.9697                     | 0.6939                 | 0.6251        | 0.5851               | 0.5697      | 0.5479           |

--&gt;
&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th scope=&#34;col&#34; style=&#34;text-align: left; background-color: #f3f4f6; color: #111;&#34;&gt;Image Type&lt;/th&gt;
      &lt;th scope=&#34;col&#34; style=&#34;text-align: right; background-color: #f3f4f6; color: #111;&#34;&gt;gemini-3-pro-image-preview&lt;/th&gt;
      &lt;th scope=&#34;col&#34; style=&#34;text-align: right; background-color: #f3f4f6; color: #111;&#34;&gt;gemini-2.5-flash-image&lt;/th&gt;
      &lt;th scope=&#34;col&#34; style=&#34;text-align: right; background-color: #f3f4f6; color: #111;&#34;&gt;gpt-image-1.5&lt;/th&gt;
      &lt;th scope=&#34;col&#34; style=&#34;text-align: right; background-color: #f3f4f6; color: #111;&#34;&gt;chatgpt-image-latest&lt;/th&gt;
      &lt;th scope=&#34;col&#34; style=&#34;text-align: right; background-color: #f3f4f6; color: #111;&#34;&gt;gpt-image-1&lt;/th&gt;
      &lt;th scope=&#34;col&#34; style=&#34;text-align: right; background-color: #f3f4f6; color: #111;&#34;&gt;gpt-image-1-mini&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th scope=&#34;row&#34; style=&#34;text-align: left; background-color: #fff; color: #000&#34;&gt;Document&lt;/th&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(37, 153, 80); color: white;&#34;&gt;89.4%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(253, 191, 113); color: black;&#34;&gt;33.6%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(226, 73, 51); color: white;&#34;&gt;14.0%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(212, 50, 44); color: white;&#34;&gt;10.0%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(203, 39, 41); color: white;&#34;&gt;7.9%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(205, 41, 41); color: white;&#34;&gt;8.3%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th scope=&#34;row&#34; style=&#34;text-align: left; background-color: #fff; color: #000&#34;&gt;Diagram&lt;/th&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(6, 114, 60); color: white;&#34;&gt;97.9%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(253, 184, 107); color: black;&#34;&gt;32.2%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(253, 185, 108); color: black;&#34;&gt;32.4%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(247, 138, 80); color: black;&#34;&gt;24.4%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(250, 153, 88); color: black;&#34;&gt;26.9%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(252, 181, 105); color: black;&#34;&gt;31.6%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th scope=&#34;row&#34; style=&#34;text-align: left; background-color: #fff; color: #000&#34;&gt;Stereogram&lt;/th&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(4, 111, 59); color: white;&#34;&gt;98.5%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(254, 212, 131); color: black;&#34;&gt;37.8%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(254, 225, 146); color: black;&#34;&gt;41.1%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(253, 194, 115); color: black;&#34;&gt;34.1%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(254, 204, 123); color: black;&#34;&gt;36.1%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(254, 219, 139); color: black;&#34;&gt;39.6%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th scope=&#34;row&#34; style=&#34;text-align: left; background-color: #fff; color: #000&#34;&gt;Scatterplot&lt;/th&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(39, 154, 81); color: white;&#34;&gt;89.1%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(59, 167, 88); color: white;&#34;&gt;85.9%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(253, 241, 167); color: black;&#34;&gt;46.3%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(254, 228, 150); color: black;&#34;&gt;41.9%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(254, 203, 123); color: black;&#34;&gt;36.0%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(254, 214, 134); color: black;&#34;&gt;38.5%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th scope=&#34;row&#34; style=&#34;text-align: left; background-color: #fff; color: #000&#34;&gt;Surreal painting&lt;/th&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(1, 105, 56); color: white;&#34;&gt;99.7%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(182, 224, 118); color: black;&#34;&gt;66.7%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(119, 196, 100); color: black;&#34;&gt;77.2%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(123, 198, 101); color: black;&#34;&gt;76.6%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(151, 210, 106); color: black;&#34;&gt;72.2%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(246, 248, 173); color: black;&#34;&gt;51.4%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th scope=&#34;row&#34; style=&#34;text-align: left; background-color: #fff; color: #000&#34;&gt;Tuvalu map&lt;/th&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(10, 122, 64); color: white;&#34;&gt;96.2%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(0, 104, 55); color: white;&#34;&gt;99.9%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(150, 210, 106); color: black;&#34;&gt;72.4%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(207, 235, 135); color: black;&#34;&gt;61.8%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(194, 229, 125); color: black;&#34;&gt;64.5%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(189, 227, 122); color: black;&#34;&gt;65.4%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th scope=&#34;row&#34; style=&#34;text-align: left; background-color: #fff; color: #000&#34;&gt;Cartoon&lt;/th&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(0, 105, 55); color: white;&#34;&gt;99.8%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(91, 183, 95); color: black;&#34;&gt;81.3%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(150, 210, 106); color: black;&#34;&gt;72.3%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(153, 211, 107); color: black;&#34;&gt;71.9%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(150, 210, 106); color: black;&#34;&gt;72.4%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(150, 210, 106); color: black;&#34;&gt;72.3%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th scope=&#34;row&#34; style=&#34;text-align: left; background-color: #fff; color: #000&#34;&gt;Human photo&lt;/th&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(2, 107, 57); color: white;&#34;&gt;99.3%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(126, 200, 102); color: black;&#34;&gt;76.1%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(80, 178, 93); color: black;&#34;&gt;82.9%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(85, 180, 94); color: black;&#34;&gt;82.2%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(140, 205, 104); color: black;&#34;&gt;74.0%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(153, 211, 107); color: black;&#34;&gt;71.8%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th scope=&#34;row&#34; style=&#34;text-align: left; background-color: #fff; color: #000&#34;&gt;Human portrait&lt;/th&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(0, 105, 55); color: white;&#34;&gt;99.9%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(67, 171, 90); color: black;&#34;&gt;84.7%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(19, 135, 71); color: white;&#34;&gt;93.4%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(36, 151, 80); color: white;&#34;&gt;89.8%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(37, 152, 80); color: white;&#34;&gt;89.5%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(121, 197, 101); color: black;&#34;&gt;77.0%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th scope=&#34;row&#34; style=&#34;text-align: left; background-color: #fff; color: #000&#34;&gt;Landscape&lt;/th&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(0, 105, 55); color: white;&#34;&gt;99.8%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(12, 124, 66); color: white;&#34;&gt;95.7%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(21, 137, 72); color: white;&#34;&gt;93.1%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(23, 139, 73); color: white;&#34;&gt;92.4%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(33, 149, 78); color: white;&#34;&gt;90.3%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(25, 141, 74); color: white;&#34;&gt;92.0%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th scope=&#34;row&#34; style=&#34;text-align: left; background-color: #fff; color: #000&#34;&gt;Overall Average&lt;/th&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(8, 118, 63); color: white;&#34;&gt;97.0%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(167, 218, 112); color: black;&#34;&gt;69.4%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(203, 233, 132); color: black;&#34;&gt;62.5%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(221, 241, 149); color: black;&#34;&gt;58.5%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(228, 243, 155); color: black;&#34;&gt;57.0%&lt;/td&gt;
      &lt;td style=&#34;text-align: right; background-color: rgb(236, 246, 164); color: black;&#34;&gt;54.8%&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;For example, take this portrait:&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://files.s-anand.net/images/2026-01-22-imagediffer-human.avif&#34;&gt;&lt;/p&gt;
&lt;p&gt;Gemini 3 Pro is nearly pixel perfect. (Red areas show differences. You can see that most areas are barely different.) Whereas GPT Image 1.5 makes more changes:&lt;/p&gt;
&lt;p&gt;The nose is smaller.&lt;br&gt;
The mouth is wider.&lt;br&gt;
The lips are thinner.&lt;br&gt;
The eyes are smaller.&lt;br&gt;
The moustache is smaller.&lt;br&gt;
The goatee is neater.&lt;br&gt;
The face is narrower.&lt;br&gt;
The skin is yellower.&lt;br&gt;
&amp;hellip; and so on.&lt;/p&gt;
&lt;p&gt;But even the best models aren&amp;rsquo;t perfect when copying diagrams and documents.&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://files.s-anand.net/images/2026-01-22-imagediffer-document.avif&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://files.s-anand.net/images/2026-01-22-imagediffer-diagram.avif&#34;&gt;&lt;/p&gt;
&lt;p&gt;The text is slightly displaced. Background writing is removed. Gemini 3 Pro actually sharpens the text. But even here, interestingly, almost every letter is copied correctly - the differences are small.&lt;/p&gt;
&lt;h3 id=&#34;what-does-this-mean&#34;&gt;What does this mean?&lt;/h3&gt;
&lt;p&gt;Clearly, Gemini 3 Pro Image Preview outperforms the others by a huge margin. Winner in 10 categories, with a 97% average. The runner up (Gemini 2.5 Flash Image) wins barely nudges it out on the Tuvalu map, and has a 69% average.&lt;/p&gt;
&lt;p&gt;Also, the models are clearly better at &lt;strong&gt;natural images&lt;/strong&gt; (human photo, portrait, landscape) than with &lt;strong&gt;symbolic images&lt;/strong&gt; (cartoon, diagram, scatterplot, document).&lt;/p&gt;
&lt;p&gt;Among the OpenAI models, GPT Image 1.5 is far better than the others, but lags far behind Gemini 3 Pro. The gap is larger for symbolic images - so &lt;strong&gt;definitely prefer the Gemini models to edit diagrams, documents, charts etc.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Also, Gemini 3 Pro Image Preview seems &lt;em&gt;quite&lt;/em&gt; different from Gemini 2.5 Flash (20% correlation)- almost a different model, whereas GPT Image 1.5 seems fairly similar to GPT Image 1 and GPT Image 1 Mini (97% correlation) - like incremental improvements to the same model.&lt;/p&gt;
&lt;!-- Source: https://chatgpt.com/c/6971a446-a108-8322-bba6-bce54360c03e --&gt;
&lt;hr&gt;
&lt;p&gt;You can try out your own images at &lt;a href=&#34;https://nitin399-maker.github.io/imagediffer/&#34;&gt;https://nitin399-maker.github.io/imagediffer/&lt;/a&gt; or run the &lt;a href=&#34;https://github.com/Nitin399-maker/imagediffer&#34;&gt;code&lt;/a&gt; yourself.&lt;/p&gt;
</description>
    </item>
    <item>
      <title>LLMs still do not locate bounding boxes well</title>
      <link>https://www.s-anand.net/blog/llms-still-do-not-locate-bounding-boxes-well/</link>
      <pubDate>Fri, 01 Nov 2024 16:20:03 +0000</pubDate>
      <guid>https://www.s-anand.net/blog/llms-still-do-not-locate-bounding-boxes-well/</guid>
      <description>&lt;p&gt;I sent an image to over a dozen LLMs that support vision, asking them:&lt;/p&gt;
&lt;blockquote class=&#34;wp-block-quote&#34;&gt;
&lt;p&gt;Detect objects in this 1280x720 px image and return their color and bounding boxes in pixels. Respond as a JSON object: {[label]: [color, x1, y1, x2, y2], …}&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;None of the models did a good-enough job. It looks like we have some time to go before LLMs become good at bounding boxes.&lt;/p&gt;
&lt;p&gt;I&#39;ve given them a subjective rating on a 1-5 scale below. &lt;/p&gt;
&lt;figure class=&#34;wp-block-table&#34;&gt;&lt;table class=&#34;has-fixed-layout&#34;&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Model&lt;/th&gt;&lt;th&gt;Positions&lt;/th&gt;&lt;th&gt;Sizes&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;gemini-1.5-flash-001&lt;/td&gt;&lt;td&gt;🟢🟢🟢🔴🔴&lt;/td&gt;&lt;td&gt;🟢🟢🟢🟢🔴&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;gemini-1.5-flash-8b&lt;/td&gt;&lt;td&gt;🟢🟢🟢🔴🔴&lt;/td&gt;&lt;td&gt;🟢🟢🟢🔴🔴&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;gemini-1.5-flash-002&lt;/td&gt;&lt;td&gt;🟢🟢🔴🔴🔴&lt;/td&gt;&lt;td&gt;🟢🟢🟢🔴🔴&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;gemini-1.5-pro-002&lt;/td&gt;&lt;td&gt;🟢🟢🟢🔴🔴&lt;/td&gt;&lt;td&gt;🟢🟢🟢🟢🔴&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;gpt-4o-mini&lt;/td&gt;&lt;td&gt;🟢🔴🔴🔴🔴&lt;/td&gt;&lt;td&gt;🟢🟢🔴🔴🔴&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;gpt-4o&lt;/td&gt;&lt;td&gt;🟢🟢🟢🟢🔴&lt;/td&gt;&lt;td&gt;🟢🟢🟢🟢🔴&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;chatgpt-4o-latest&lt;/td&gt;&lt;td&gt;🟢🟢🟢🟢🔴&lt;/td&gt;&lt;td&gt;🟢🟢🟢🟢🔴&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;claude-3-haiku-20240307&lt;/td&gt;&lt;td&gt;🟢🔴🔴🔴🔴&lt;/td&gt;&lt;td&gt;🟢🟢🔴🔴🔴&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;claude-3=5-sonnet-20241022&lt;/td&gt;&lt;td&gt;🟢🟢🟢🔴🔴&lt;/td&gt;&lt;td&gt;🟢🟢🟢🔴🔴&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;llama-3.2-11b-vision-preview&lt;/td&gt;&lt;td&gt;🔴🔴🔴🔴🔴&lt;/td&gt;&lt;td&gt;🔴🔴🔴🔴🔴&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;llama-3.2-90b-vision-preview&lt;/td&gt;&lt;td&gt;🟢🟢🟢🔴🔴&lt;/td&gt;&lt;td&gt;🟢🟢🟢🔴🔴&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;qwen-2-vl-72b-instruct&lt;/td&gt;&lt;td&gt;🟢🟢🟢🔴🔴&lt;/td&gt;&lt;td&gt;🟢🟢🔴🔴🔴&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;pixtral-12b&lt;/td&gt;&lt;td&gt;🟢🟢🔴🔴🔴&lt;/td&gt;&lt;td&gt;🟢🟢🟢🔴🔴&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/figure&gt;
&lt;p&gt;I used an &lt;a href=&#34;https://tools.s-anand.net/llmboundingbox/&#34;&gt;app I built for this&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Here is the original image along with the individual results.&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/shapes-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/gemini-1.5-flash-8b-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/gemini-1.5-flash-001-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/gemini-1.5-flash-002-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/gemini-1.5-pro-002-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/gpt-4o-mini-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/gpt-4o-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/chatgpt-4o-latest-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/claude-3-haiku-20240307-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/claude-3-5-sonnet-20241022-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/llama-3.2-11b-vision-preview-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/llama-3.2-90b-vision-preview-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/pixtral-12b-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/qwen-2-vl-72b-instruct-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;h4 class=&#34;wp-block-heading&#34;&gt;Update&lt;/h4&gt;
&lt;p&gt;Adding gridlines with labeled axes helps the LLMs. (Thanks &lt;a href=&#34;https://www.linkedin.com/feed/update/urn:li:ugcPost:7258152478255243264?commentUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A7258152478255243264%2C7258184350876262402%29&amp;amp;dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287258184350876262402%2Curn%3Ali%3AugcPost%3A7258152478255243264%29&#34;&gt;@Bijan Mishra&lt;/a&gt;.) Here are a few examples:&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/chatgpt-4o-latest-1-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/gemini-1.5-flash-002-1-1024x576.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/claude-3-5-sonnet-20241022-1-1024x576.webp&#34;&gt;&lt;/p&gt;
</description>
    </item>
    <item>
      <title>Google may acquire Riya</title>
      <link>https://www.s-anand.net/blog/google-may-acquire-riya/</link>
      <pubDate>Thu, 17 Nov 2005 12:00:00 +0000</pubDate>
      <guid>https://www.s-anand.net/blog/google-may-acquire-riya/</guid>
      <description>&lt;p&gt;&lt;a href=&#34;http://blog.searchenginewatch.com/blog/051117-154647&#34;&gt;Google may acquire Riya&lt;/a&gt;. &lt;a href=&#34;http://www.riya.com/&#34;&gt;Riya&lt;/a&gt; can recognise faces and lettering in your pictures.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id=&#34;comments&#34;&gt;Comments&lt;/h2&gt;
&lt;!-- wp-comments-start --&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Sai&lt;/strong&gt; &lt;em&gt;8 Dec 2005 11:50 pm&lt;/em&gt;:
Hi, do post the rest of the tips for excel when you get the time. Good Stuff.&lt;/li&gt;
&lt;/ul&gt;
&lt;!-- wp-comments-end --&gt;
</description>
    </item>
    <item>
      <title>Identify location by photographs</title>
      <link>https://www.s-anand.net/blog/identify-location-by-photographs/</link>
      <pubDate>Wed, 14 Apr 2004 12:00:00 +0000</pubDate>
      <guid>https://www.s-anand.net/blog/identify-location-by-photographs/</guid>
      <description>&lt;p&gt;There&amp;rsquo;s software that can &lt;a href=&#34;http://www.newscientist.com/news/news.jsp?id=ns99994857&#34;&gt;locate where you are&lt;/a&gt; using a photograph of buildings.&lt;/p&gt;
</description>
    </item>
    <item>
      <title>Cats</title>
      <link>https://www.s-anand.net/blog/cats/</link>
      <pubDate>Tue, 26 Mar 2002 12:00:00 +0000</pubDate>
      <guid>https://www.s-anand.net/blog/cats/</guid>
      <description>&lt;p&gt;Amazing, what some people try and do with &lt;a href=&#34;http://www.quantumpicture.com/Flo_Control/flo_control.htm&#34;&gt;cats&lt;/a&gt;. And image recognition.&lt;/p&gt;
</description>
    </item>
  </channel>
</rss>
