LLMs still do not locate bounding boxes well

Fri, 01 Nov 2024 16:20:03 +0000

I sent an image to over a dozen LLMs that support vision, asking them:

Detect objects in this 1280x720 px image and return their color and bounding boxes in pixels. Respond as a JSON object: {[label]: [color, x1, y1, x2, y2], …}

None of the models did a good-enough job. It looks like we have some time to go before LLMs become good at bounding boxes.

I've given them a subjective rating on a 1-5 scale below.

Model	Positions	Sizes
gemini-1.5-flash-001	🟢🟢🟢🔴🔴	🟢🟢🟢🟢🔴
gemini-1.5-flash-8b	🟢🟢🟢🔴🔴	🟢🟢🟢🔴🔴
gemini-1.5-flash-002	🟢🟢🔴🔴🔴	🟢🟢🟢🔴🔴
gemini-1.5-pro-002	🟢🟢🟢🔴🔴	🟢🟢🟢🟢🔴
gpt-4o-mini	🟢🔴🔴🔴🔴	🟢🟢🔴🔴🔴
gpt-4o	🟢🟢🟢🟢🔴	🟢🟢🟢🟢🔴
chatgpt-4o-latest	🟢🟢🟢🟢🔴	🟢🟢🟢🟢🔴
claude-3-haiku-20240307	🟢🔴🔴🔴🔴	🟢🟢🔴🔴🔴
claude-3=5-sonnet-20241022	🟢🟢🟢🔴🔴	🟢🟢🟢🔴🔴
llama-3.2-11b-vision-preview	🔴🔴🔴🔴🔴	🔴🔴🔴🔴🔴
llama-3.2-90b-vision-preview	🟢🟢🟢🔴🔴	🟢🟢🟢🔴🔴
qwen-2-vl-72b-instruct	🟢🟢🟢🔴🔴	🟢🟢🔴🔴🔴
pixtral-12b	🟢🟢🔴🔴🔴	🟢🟢🟢🔴🔴

I used an app I built for this.

Here is the original image along with the individual results.

Update

Adding gridlines with labeled axes helps the LLMs. (Thanks @Bijan Mishra.) Here are a few examples:

Google may acquire Riya

Thu, 17 Nov 2005 12:00:00 +0000

Google may acquire Riya. Riya can recognise faces and lettering in your pictures.

Comments

Sai 8 Dec 2005 11:50 pm: Hi, do post the rest of the tips for excel when you get the time. Good Stuff.

Identify location by photographs

Wed, 14 Apr 2004 12:00:00 +0000

There’s software that can locate where you are using a photograph of buildings.

Cats

Tue, 26 Mar 2002 12:00:00 +0000

Amazing, what some people try and do with cats. And image recognition.