There are 4 frontier #LLMs today. No other (popular) model beats them on BOTH cost and quality.
- llama-3-8b-instruct
- claude-3-haiku-20240307
- llama-3-70b-instruct
- gpt-4o-2024-05-13
This list changes rapidly. But in practice, it means there’s little reason to use any other LLM. They beat every other model on cost and quality (measured by the LMSYS Arena ELO score.)
I opened Straive + Gramener’s keynote yesterday at marcus evans Group’s Digitech forum with this. Strange that this is not well known. Especially as switching from GPT-4 to Claude 3 Haiku can shrink a $1.2 million Gen AI budget to just $10K.
See the interactive version at https://sanand0.github.io/llmpricing/
- 10 May 2024: mistral-7b-instruct-v0.2 was dropped since llama-3-8b-instruct is available for cheaper on Replicate.
- 19 May 2024: gemini-1.5-pro-api-0409-preview and gpt-4-turbo-2024-04-09 were dropped since gpt-4o-2024-05-13 is half the price at similar quality
