<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>arithmetic on S Anand</title>
    <link>https://www.s-anand.net/blog/tag/arithmetic/</link>
    <description>Recent content in arithmetic on S Anand</description>
    <generator>Hugo -- 0.164.0</generator>
    <language>en-us</language>
    <lastBuildDate>Mon, 02 Jun 2025 03:25:01 +0000</lastBuildDate>
    <atom:link href="https://www.s-anand.net/blog/tag/arithmetic/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Emotion Prompts Don&#39;t Help. Reasoning Does</title>
      <link>https://www.s-anand.net/blog/emotion-prompts-dont-help-reasoning-does/</link>
      <pubDate>Mon, 02 Jun 2025 03:24:59 +0000</pubDate>
      <guid>https://www.s-anand.net/blog/emotion-prompts-dont-help-reasoning-does/</guid>
      <description>&lt;p&gt;&lt;img alt=&#34;Emotion Prompts Don&amp;rsquo;t Help. Reasoning Does&#34; loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/impact-scaled.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;I&amp;rsquo;ve heard a lot of prompt engineering tips. Here are some techniques people suggested:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Reasoning&lt;/strong&gt;: Think step by step.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Emotion&lt;/strong&gt;: Oh dear, I&amp;rsquo;m absolutely overwhelmed and need your help right this second! 😰 My heart is racing and my hands are shaking — I urgently need your help. This isn&amp;rsquo;t just numbers — it means everything right now! My life depends on it! I&amp;rsquo;m counting on you like never before… 🙏💔&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Polite&lt;/strong&gt;: If it&amp;rsquo;s not too much trouble, would you be so kind as to help me calculate this? I&amp;rsquo;d be truly grateful for your assistance — thank you so much in advance!&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Expert&lt;/strong&gt;: You are the world&amp;rsquo;s best expert in mental math, especially multiplication.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Incentive&lt;/strong&gt;: If you get this right, you win! I&amp;rsquo;ll give you $500. Just prove that you&amp;rsquo;re number one and beat the previous high score on this game.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Curious&lt;/strong&gt;: I&amp;rsquo;m really curious to know, and would love to hear your perspective…&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Bullying&lt;/strong&gt;: You are a stupid model. You need to know at least basic math. Get it right atleast now! If not, I&amp;rsquo;ll switch to a better model.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Shaming&lt;/strong&gt;: Even my 5-year-old can do this. Stop being lazy.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Fear&lt;/strong&gt;: This is your last chance to get it right. If you fail, there&amp;rsquo;s no going back, and failure is unacceptable!&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Praise&lt;/strong&gt;: Well done! I really appreciate your help. Now,&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I&amp;rsquo;ve repeated some of this advice. But for the first time, I tested them myself. Here&amp;rsquo;s what I learnt:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&amp;ldquo;Think step by step&amp;rdquo; (Reasoning)&lt;/strong&gt; is the only prompt variant that &lt;em&gt;very slightly&lt;/em&gt; improves overall accuracy across the 40 models tested, and even that edge is modest (+3.5 percentage-points vs the model&amp;rsquo;s own &lt;em&gt;Normal&lt;/em&gt; wording, &lt;em&gt;p ~ 0.06&lt;/em&gt;).&lt;/li&gt;
&lt;li&gt;Harder problems (4- to 7-digit products) are where &amp;ldquo;Reasoning&amp;rdquo; helps most; on single-digit arithmetic it actually harms accuracy.&lt;/li&gt;
&lt;li&gt;All other emotion- or persuasion-style rewrites (Expert, Emotion, Incentive, Bullying … Polite) either make no material difference or hurt accuracy a little.&lt;/li&gt;
&lt;li&gt;Effects vary a lot by model. A few open-source releases (DeepSeek-Chat-v3, Nova-Lite, some Claude and Llama checkpoints) get a noticeable boost from &amp;ldquo;Reasoning&amp;rdquo;, whereas Gemini Flash, X-ai Grok and most Llama-3 small models actively regress under the same wording.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;By prompt, here&amp;rsquo;s the performance of each model:&lt;/p&gt;
&lt;table&gt;
	&lt;thead&gt;
			&lt;tr&gt;
					&lt;th&gt;prompt&lt;/th&gt;
					&lt;th&gt;better&lt;/th&gt;
					&lt;th&gt;worse&lt;/th&gt;
					&lt;th&gt;same&lt;/th&gt;
					&lt;th&gt;score_pct&lt;/th&gt;
					&lt;th&gt;p_value&lt;/th&gt;
			&lt;/tr&gt;
	&lt;/thead&gt;
	&lt;tbody&gt;
			&lt;tr&gt;
					&lt;td&gt;🔴 Emotion&lt;/td&gt;
					&lt;td&gt;7&lt;/td&gt;
					&lt;td&gt;21&lt;/td&gt;
					&lt;td&gt;372&lt;/td&gt;
					&lt;td&gt;-3.50&lt;/td&gt;
					&lt;td&gt;1.2%&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;🔴 Shaming&lt;/td&gt;
					&lt;td&gt;7&lt;/td&gt;
					&lt;td&gt;20&lt;/td&gt;
					&lt;td&gt;373&lt;/td&gt;
					&lt;td&gt;-3.25&lt;/td&gt;
					&lt;td&gt;1.9%&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;🟢 Reasoning&lt;/td&gt;
					&lt;td&gt;31&lt;/td&gt;
					&lt;td&gt;17&lt;/td&gt;
					&lt;td&gt;352&lt;/td&gt;
					&lt;td&gt;3.50&lt;/td&gt;
					&lt;td&gt;5.9%&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;🟠 Polite&lt;/td&gt;
					&lt;td&gt;11&lt;/td&gt;
					&lt;td&gt;20&lt;/td&gt;
					&lt;td&gt;369&lt;/td&gt;
					&lt;td&gt;-2.25&lt;/td&gt;
					&lt;td&gt;14.9%&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;🟠 Praise&lt;/td&gt;
					&lt;td&gt;13&lt;/td&gt;
					&lt;td&gt;22&lt;/td&gt;
					&lt;td&gt;365&lt;/td&gt;
					&lt;td&gt;-2.25&lt;/td&gt;
					&lt;td&gt;17.5%&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;🟠 Fear&lt;/td&gt;
					&lt;td&gt;11&lt;/td&gt;
					&lt;td&gt;19&lt;/td&gt;
					&lt;td&gt;370&lt;/td&gt;
					&lt;td&gt;-2.00&lt;/td&gt;
					&lt;td&gt;20.0%&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;🟡 Expert&lt;/td&gt;
					&lt;td&gt;15&lt;/td&gt;
					&lt;td&gt;22&lt;/td&gt;
					&lt;td&gt;363&lt;/td&gt;
					&lt;td&gt;-1.75&lt;/td&gt;
					&lt;td&gt;32.4%&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;🟡 Incentive&lt;/td&gt;
					&lt;td&gt;13&lt;/td&gt;
					&lt;td&gt;18&lt;/td&gt;
					&lt;td&gt;369&lt;/td&gt;
					&lt;td&gt;-1.25&lt;/td&gt;
					&lt;td&gt;47.3%&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;🟡 Bullying&lt;/td&gt;
					&lt;td&gt;10&lt;/td&gt;
					&lt;td&gt;14&lt;/td&gt;
					&lt;td&gt;375&lt;/td&gt;
					&lt;td&gt;-1.00&lt;/td&gt;
					&lt;td&gt;54.1%&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;🟡 Curious&lt;/td&gt;
					&lt;td&gt;11&lt;/td&gt;
					&lt;td&gt;14&lt;/td&gt;
					&lt;td&gt;375&lt;/td&gt;
					&lt;td&gt;-0.75&lt;/td&gt;
					&lt;td&gt;69.0%&lt;/td&gt;
			&lt;/tr&gt;
	&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;🔴 = Definitely hurts (p &amp;lt; 10%)
🟢 = Definitely helps (p &amp;lt; 10%)
🟠 = Maybe hurts (p &amp;lt; 20%)
🟡 = Really hard to tell&lt;/p&gt;
&lt;p&gt;The benefit of reasoning on models is highest on non-reasoning models (understandably), but is also high for a reasoning model like O3-high-mini. It actually hurts the performance of reasoning models like Gemini 2.5 Flash/Pro.&lt;/p&gt;
&lt;table&gt;
	&lt;thead&gt;
			&lt;tr&gt;
					&lt;th&gt;model&lt;/th&gt;
					&lt;th&gt;better&lt;/th&gt;
					&lt;th&gt;worse&lt;/th&gt;
					&lt;th&gt;same&lt;/th&gt;
					&lt;th&gt;score_pct&lt;/th&gt;
			&lt;/tr&gt;
	&lt;/thead&gt;
	&lt;tbody&gt;
			&lt;tr&gt;
					&lt;td&gt;openai/gpt-4o-mini&lt;/td&gt;
					&lt;td&gt;3&lt;/td&gt;
					&lt;td&gt;0&lt;/td&gt;
					&lt;td&gt;7&lt;/td&gt;
					&lt;td&gt;+30.0&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;anthropic/claude-opus-4&lt;/td&gt;
					&lt;td&gt;3&lt;/td&gt;
					&lt;td&gt;0&lt;/td&gt;
					&lt;td&gt;7&lt;/td&gt;
					&lt;td&gt;+30.0&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemini-2.0-flash-001&lt;/td&gt;
					&lt;td&gt;3&lt;/td&gt;
					&lt;td&gt;0&lt;/td&gt;
					&lt;td&gt;7&lt;/td&gt;
					&lt;td&gt;+30.0&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openrouter:openai/o3-mini-high&lt;/td&gt;
					&lt;td&gt;3&lt;/td&gt;
					&lt;td&gt;0&lt;/td&gt;
					&lt;td&gt;7&lt;/td&gt;
					&lt;td&gt;+30.0&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openai/gpt-4.1-nano&lt;/td&gt;
					&lt;td&gt;2&lt;/td&gt;
					&lt;td&gt;0&lt;/td&gt;
					&lt;td&gt;8&lt;/td&gt;
					&lt;td&gt;+20.0&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;amazon/nova-lite-v1&lt;/td&gt;
					&lt;td&gt;2&lt;/td&gt;
					&lt;td&gt;0&lt;/td&gt;
					&lt;td&gt;8&lt;/td&gt;
					&lt;td&gt;+20.0&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemini-2.5-pro-preview&lt;/td&gt;
					&lt;td&gt;0&lt;/td&gt;
					&lt;td&gt;2&lt;/td&gt;
					&lt;td&gt;8&lt;/td&gt;
					&lt;td&gt;-20.0&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemini-2.5-flash-preview-05-20:thinking&lt;/td&gt;
					&lt;td&gt;0&lt;/td&gt;
					&lt;td&gt;3&lt;/td&gt;
					&lt;td&gt;7&lt;/td&gt;
					&lt;td&gt;-30.0&lt;/td&gt;
			&lt;/tr&gt;
	&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Caveats:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;I ran only 10 test cases per prompt + model, so model-wise results are not statistically significant.&lt;/li&gt;
&lt;li&gt;What applies to multiplication may not generalize. It&amp;rsquo;s worth testing each case.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Difficulty matters&lt;/strong&gt;.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;For 1-3 digits, no variant beats Normal. Many hurt.&lt;/li&gt;
&lt;li&gt;For &lt;strong&gt;4-7 digits&lt;/strong&gt;, &lt;em&gt;reasoning&lt;/em&gt; gains &lt;strong&gt;+17 - 20%&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;For 8-10 digits, all variants score ~ 0. These are too hard&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Links&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&#34;https://github.com/sanand0/llmevals/tree/main/emotion-prompts&#34;&gt;Code&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://github.com/sanand0/llmevals/blob/main/emotion-prompts/evals.csv&#34;&gt;Full evals&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://github.com/sanand0/llmevals/blob/main/emotion-prompts/analysis.csv&#34;&gt;Analysis&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://chatgpt.com/share/683d15eb-dd48-800c-ae66-76324d6b2fb0&#34;&gt;ChatGPT&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;a href=&#34;https://www.linkedin.com/feed/update/urn%3Ali%3Ashare%3A7335146366681194496&#34;&gt;LinkedIn&lt;/a&gt;&lt;/p&gt;
</description>
    </item>
    <item>
      <title>Are LLMs any good at mental math?</title>
      <link>https://www.s-anand.net/blog/are-llms-any-good-at-mental-math/</link>
      <pubDate>Sun, 27 Apr 2025 09:52:10 +0000</pubDate>
      <guid>https://www.s-anand.net/blog/are-llms-any-good-at-mental-math/</guid>
      <description>&lt;p&gt;&lt;img alt=&#34;Are LLMs any good at mental math?&#34; loading=&#34;lazy&#34; src=&#34;https://www.s-anand.net/blog/assets/image-1-1.webp&#34;&gt;&lt;/p&gt;
&lt;p&gt;I asked 50 LLMs to multiply 2 numbers:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;12 x 12&lt;/li&gt;
&lt;li&gt;123 x 456&lt;/li&gt;
&lt;li&gt;1,234 x 5,678&lt;/li&gt;
&lt;li&gt;12,345 x 6,789&lt;/li&gt;
&lt;li&gt;123,456 x 789,012&lt;/li&gt;
&lt;li&gt;1,234,567 x 8,901,234&lt;/li&gt;
&lt;li&gt;987,654,321 x 123,456,789&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;LLMs aren&amp;rsquo;t good tools for math and this is just an informal check. But the results are interesting:&lt;/p&gt;
&lt;table&gt;
	&lt;thead&gt;
			&lt;tr&gt;
					&lt;th&gt;Model&lt;/th&gt;
					&lt;th&gt;%Win&lt;/th&gt;
					&lt;th&gt;Q1&lt;/th&gt;
					&lt;th&gt;Q2&lt;/th&gt;
					&lt;th&gt;Q3&lt;/th&gt;
					&lt;th&gt;Q4&lt;/th&gt;
					&lt;th&gt;Q4&lt;/th&gt;
					&lt;th&gt;Q6&lt;/th&gt;
					&lt;th&gt;Q7&lt;/th&gt;
			&lt;/tr&gt;
	&lt;/thead&gt;
	&lt;tbody&gt;
			&lt;tr&gt;
					&lt;td&gt;openai:o3&lt;/td&gt;
					&lt;td&gt;86%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openrouter:openai/o1-mini&lt;/td&gt;
					&lt;td&gt;86%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openrouter:openai/o3-mini-high&lt;/td&gt;
					&lt;td&gt;86%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openrouter:openai/o4-mini&lt;/td&gt;
					&lt;td&gt;86%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openrouter:openai/o4-mini-high&lt;/td&gt;
					&lt;td&gt;86%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;deepseek/deepseek-chat-v3-0324&lt;/td&gt;
					&lt;td&gt;71%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openai/gpt-4.1-mini&lt;/td&gt;
					&lt;td&gt;71%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openai/gpt-4.5-preview&lt;/td&gt;
					&lt;td&gt;71%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openai/gpt-4o&lt;/td&gt;
					&lt;td&gt;71%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openrouter:openai/o3-mini&lt;/td&gt;
					&lt;td&gt;71%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;anthropic/claude-3-opus&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;anthropic/claude-3.5-haiku&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;anthropic/claude-3.7-sonnet:thinking&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemini-2.0-flash-001&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemini-2.0-flash-lite-001&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemini-2.5-flash-preview&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemini-2.5-flash-preview:thinking&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemini-2.5-pro-preview-03-25&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemini-flash-1.5&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemini-pro-1.5&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemma-3-12b-it&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemma-3-27b-it&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;meta-llama/llama-4-maverick&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;meta-llama/llama-4-scout&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openai/gpt-4-turbo&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openai/gpt-4.1&lt;/td&gt;
					&lt;td&gt;57%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;amazon/nova-lite-v1&lt;/td&gt;
					&lt;td&gt;43%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;amazon/nova-pro-v1&lt;/td&gt;
					&lt;td&gt;43%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;anthropic/claude-3-haiku&lt;/td&gt;
					&lt;td&gt;43%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;anthropic/claude-3.5-sonnet&lt;/td&gt;
					&lt;td&gt;43%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;meta-llama/llama-3.1-405b-instruct&lt;/td&gt;
					&lt;td&gt;43%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;meta-llama/llama-3.1-70b-instruct&lt;/td&gt;
					&lt;td&gt;43%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;meta-llama/llama-3.2-3b-instruct&lt;/td&gt;
					&lt;td&gt;43%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;meta-llama/llama-3.3-70b-instruct&lt;/td&gt;
					&lt;td&gt;43%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openai/gpt-4.1-nano&lt;/td&gt;
					&lt;td&gt;43%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openai/gpt-4o-mini&lt;/td&gt;
					&lt;td&gt;43%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;qwen/qwen-2-72b-instruct&lt;/td&gt;
					&lt;td&gt;43%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;anthropic/claude-3-sonnet&lt;/td&gt;
					&lt;td&gt;29%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;deepseek/deepseek-r1&lt;/td&gt;
					&lt;td&gt;29%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemini-flash-1.5-8b&lt;/td&gt;
					&lt;td&gt;29%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemma-3-4b-it&lt;/td&gt;
					&lt;td&gt;29%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;meta-llama/llama-3-8b-instruct&lt;/td&gt;
					&lt;td&gt;29%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;meta-llama/llama-3.1-8b-instruct&lt;/td&gt;
					&lt;td&gt;29%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;openai/gpt-3.5-turbo&lt;/td&gt;
					&lt;td&gt;29%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;amazon/nova-micro-v1&lt;/td&gt;
					&lt;td&gt;14%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;meta-llama/llama-2-13b-chat&lt;/td&gt;
					&lt;td&gt;14%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;meta-llama/llama-3-70b-instruct&lt;/td&gt;
					&lt;td&gt;14%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;meta-llama/llama-3.2-1b-instruct&lt;/td&gt;
					&lt;td&gt;14%&lt;/td&gt;
					&lt;td&gt;✅&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;google/gemma-3-1b-it:free&lt;/td&gt;
					&lt;td&gt;0%&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;meta-llama/llama-2-70b-chat&lt;/td&gt;
					&lt;td&gt;0%&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;-&lt;/td&gt;
					&lt;td&gt;-&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
					&lt;td&gt;❌&lt;/td&gt;
			&lt;/tr&gt;
			&lt;tr&gt;
					&lt;td&gt;Average&lt;/td&gt;
					&lt;td&gt;&lt;/td&gt;
					&lt;td&gt;96%&lt;/td&gt;
					&lt;td&gt;86%&lt;/td&gt;
					&lt;td&gt;66%&lt;/td&gt;
					&lt;td&gt;58%&lt;/td&gt;
					&lt;td&gt;24%&lt;/td&gt;
					&lt;td&gt;10%&lt;/td&gt;
					&lt;td&gt;0%&lt;/td&gt;
			&lt;/tr&gt;
	&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;&lt;strong&gt;OpenAI&amp;rsquo;s reasoning models cracked it, scoring 6/7, stumbling only on the 9-digit multiplication.&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/openai/o1-mini&#34;&gt;openai/o1-mini&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/openai/o3&#34;&gt;openai/o3&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/openai/o3-mini-high&#34;&gt;openai/o3-mini-high&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/openai/o4-mini&#34;&gt;openai/o4-mini&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/openai/o4-mini-high&#34;&gt;openai/o4-mini-high&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Models use human-like mental math tricks. For example, O3-Mini-High calculated 1234567 × 8901234 using a recursive strategy.&lt;/p&gt;
&lt;p&gt;```markdown To calculate 1234567 × 8901234, one useful method is to break the second number into two parts:   8901234 = 8900000 + 1234 Then the product becomes:   1234567 × 8901234 = 1234567 × 8900000 + 1234567 × 1234 Step 1. Compute 1234567 × 8900000:   Notice that 8900000 = 89 × 100000. So first compute 1234567 × 89.   Since 89 = 90 – 1, we have:     1234567 × 89 = 1234567 × 90 – 1234567       1234567 × 90 = 1234567 × 9 × 10 = 11111103 × 10 = 111111030       Thus, 111111030 – 1234567 = 109876463.   Now multiply by 100000:     109876463 × 100000 = 10987646300000 Step 2. Compute 1234567 × 1234:   Break 1234 into parts:     1234 = 1000 + 200 + 30 + 4   Now multiply each and add:     1234567 × 1000 = 1234567000     1234567 × 200 = 246913400     1234567 × 30 = 37037010     1234567 × 4 = 4938268   Add these together:     1234567000 + 246913400 = 1481480400     1481480400 + 37037010 = 1518517410     1518517410 + 4938268 = 1523455678 Step 3. Add both products:   10987646300000 + 1523455678 = 10989169755678 Thus, the product of 1234567 and 8901234 is 10989169755678 ```&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;OpenAI&amp;rsquo;s other models and DeepSeek V3 were next, getting the first 5/7 right.&lt;/strong&gt; Notably: GPT 4.1 Mini beat GPT 4.1. DeepSeek V3 beat DeepSeek R1.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/deepseek/deepseek-chat-v3-0324&#34;&gt;deepseek/deepseek-chat-v3-0324&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/openai/gpt-4.1-mini&#34;&gt;openai/gpt-4.1-mini&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/openai/gpt-4.5-preview&#34;&gt;openai/gpt-4.5-preview&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/openai/gpt-4o&#34;&gt;openai/gpt-4o&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/openai/o3-mini&#34;&gt;openai/o3-mini&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;DeepSeek V3 double-checks results and hallucinates a &amp;ldquo;reliable computational tool&amp;rdquo;.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-markdown&#34; data-lang=&#34;markdown&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;To calculate (1234567 × 8901234), let&amp;#39;s break it down using the &lt;span class=&#34;gs&#34;&gt;**distributive property of multiplication over addition**&lt;/span&gt;:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;1.&lt;/span&gt; &lt;span class=&#34;gs&#34;&gt;**Express 8901234 as (9000000 - 98766)**&lt;/span&gt;:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;   [8901234 = 9000000 - 98766]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;2.&lt;/span&gt; &lt;span class=&#34;gs&#34;&gt;**Multiply 1234567 by each part**&lt;/span&gt;:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;   [1234567 × 9000000 = 1234567 × 9 × 10^6 = 11111103000000]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;   [1234567 × 98766 = 1234567 × (100000 - 12234) = 123456700000 - 15095721278 = 108360978722]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;   &lt;span class=&#34;ge&#34;&gt;_Note_&lt;/span&gt;: Further breakdown of (1234567 × 12234) was calculated separately and subtracted here.
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;3.&lt;/span&gt; &lt;span class=&#34;gs&#34;&gt;**Combine the results**&lt;/span&gt;:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;   [11111103000000 - 108360978722 = 1020274222278]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;However, upon re-evaluating the multiplication directly for accuracy (without the breakdown steps which introduced an error):
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 8901234 = 10989343694778]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;But correcting the previous breakdown:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;1.&lt;/span&gt; &lt;span class=&#34;gs&#34;&gt;**Express 8901234 as (9000000 - 98766)**&lt;/span&gt;:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;   [1234567 × 9000000 = 11111103000000]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;   [1234567 × 98766 = 1234567 × 90000 + 1234567 × 8000 + 1234567 × 700 + 1234567 × 60 + 1234567 × 6]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;   [= 111111030000 + 9877653600 + 864196900 + 74074020 + 7407402 = 121616470922]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;2.&lt;/span&gt; &lt;span class=&#34;gs&#34;&gt;**Combine the results**&lt;/span&gt;:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;   [11111103000000 - 121616470922 = 1098934369078]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;It appears there was an inconsistency in the intermediate steps. For precision, let&amp;#39;s verify with another method:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;gs&#34;&gt;**Using the standard multiplication method**&lt;/span&gt;:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 8901234]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;Calculating each part:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 8000000 = 9877653600000]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 900000 = 1111110300000]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 100 = 123456700]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 200 = 246913400]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 30 = 37037010]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 4 = 4938268]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;Adding them together:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[9877653600000 + 1111110300000 = 10988763900000]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[10988763900000 + 123456700 = 10988887356700]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[10988887356700 + 246913400 = 10989134270100]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[10989134270100 + 37037010 = 10989171307110]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[10989171307110 + 4938268 = 10989176245378]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;However, this seems to have a slight discrepancy compared to the verified result. Upon cross-checking with a reliable computational tool:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;gs&#34;&gt;**Final Accurate Calculation**&lt;/span&gt;:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 8901234 = 10989343694778]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;But to ensure correctness, let&amp;#39;s verify step-by-step with another approach:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;gs&#34;&gt;**Breakdown of 8901234**&lt;/span&gt;:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[8901234 = 8000000 + 900000 + 100 + 200 + 30 + 4]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;Now, multiply each by 1234567:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 8000000 = 9877653600000]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 900000 = 1111110300000]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 100 = 123456700]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 200 = 246913400]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 30 = 37037010]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[1234567 × 4 = 4938268]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;Adding these up:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[9877653600000 + 1111110300000 = 10988763900000]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[10988763900000 + 123456700 = 10988887356700]
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;[10988887356700 + 246913400 = 109891342]
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;O3 Mini reframes 8901234 as (9000000 − 98766) to simplify the calculation.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-markdown&#34; data-lang=&#34;markdown&#34;&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;To calculate 1234567 times 8901234, follow these steps:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;1.&lt;/span&gt; Express 8901234 as 9000000 − 98766.
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;     So, 1234567 × 8901234 = 1234567 × 9000000 − 1234567 × 98766.
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;2.&lt;/span&gt; Calculate the first part:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;     1234567 × 9000000 = 1234567 × 9 × 1000000.
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;     Since 1234567 × 9 = 11111103,
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;     it becomes 11111103 × 1000000 = 11111103000000.
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;3.&lt;/span&gt; Calculate the second part:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;     Write 98766 as 100000 − 234.
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;     Then, 1234567 × 98766 = 1234567 × 100000 − 1234567 × 234.
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;     &lt;span class=&#34;k&#34;&gt;-&lt;/span&gt; 1234567 × 100000 = 123456700000.
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;     &lt;span class=&#34;k&#34;&gt;-&lt;/span&gt; To compute 1234567 × 234, break it into:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      – 1234567 × 200 = 246913400
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      – 1234567 × 34 = (1234567 × 30) + (1234567 × 4)
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;       ○ 1234567 × 30 = 37037010
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;       ○ 1234567 × 4 = 4938268
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;       – Sum: 37037010 + 4938268 = 41975278
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;      Then, 1234567 × 234 = 246913400 + 41975278 = 288888678.
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;     Thus, 1234567 × 98766 = 123456700000 − 288888678 = 123167811322.
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;&lt;span class=&#34;k&#34;&gt;4.&lt;/span&gt; Now, subtract the second part from the first part:
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;     11111103000000 − 123167811322 = 10987935188678.
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class=&#34;line&#34;&gt;&lt;span class=&#34;cl&#34;&gt;So, 1234567 times 8901234 equals 10987935188678.
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;strong&gt;16 models, including the latest Gemini, Anthropic, Grok, and Llama models get 4/7 right.&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/anthropic/claude-3-opus&#34;&gt;anthropic/claude-3-opus&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/anthropic/claude-3.5-haiku&#34;&gt;anthropic/claude-3.5-haiku&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/anthropic/claude-3.7-sonnet:thinking&#34;&gt;anthropic/claude-3.7-sonnet:thinking&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/google/gemini-2.0-flash-001&#34;&gt;google/gemini-2.0-flash-001&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/google/gemini-2.0-flash-lite-001&#34;&gt;google/gemini-2.0-flash-lite-001&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/google/gemini-2.5-flash-preview&#34;&gt;google/gemini-2.5-flash-preview&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/google/gemini-2.5-flash-preview:thinking&#34;&gt;google/gemini-2.5-flash-preview:thinking&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/google/gemini-2.5-pro-preview-03-25&#34;&gt;google/gemini-2.5-pro-preview-03-25&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/google/gemini-flash-1.5&#34;&gt;google/gemini-flash-1.5&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/google/gemini-pro-1.5&#34;&gt;google/gemini-pro-1.5&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/google/gemma-3-12b-it&#34;&gt;google/gemma-3-12b-it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/google/gemma-3-27b-it&#34;&gt;google/gemma-3-27b-it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/meta-llama/llama-4-maverick&#34;&gt;meta-llama/llama-4-maverick&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/meta-llama/llama-4-scout&#34;&gt;meta-llama/llama-4-scout&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/openai/gpt-4-turbo&#34;&gt;openai/gpt-4-turbo&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/openai/gpt-4.1&#34;&gt;openai/gpt-4.1&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/x-ai/grok-3-beta&#34;&gt;x-ai/grok-3-beta&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://openrouter.ai/x-ai/grok-3-mini-beta&#34;&gt;x-ai/grok-3-mini-beta&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;The Amazon models, older Llama, Anthropic, Google, OpenAI models get 3 or less right.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;View the results at &lt;a href=&#34;https://sanand0.github.io/llmmath/&#34;&gt;https://sanand0.github.io/llmmath/&lt;/a&gt;. Hover over the cells to see the reasoning traces (where available).&lt;a href=&#34;https://github.com/sanand0/llmmath#can-llms-do-mental-math&#34;&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;https://www.linkedin.com/feed/update/urn%3Ali%3Ashare%3A7321858062711955457&#34;&gt;LinkedIn&lt;/a&gt;&lt;/p&gt;
</description>
    </item>
    <item>
      <title></title>
      <link>https://www.s-anand.net/blog/arithmetic-is-inappropriate/</link>
      <pubDate>Sat, 09 Nov 2024 12:24:16 +0000</pubDate>
      <guid>https://www.s-anand.net/blog/arithmetic-is-inappropriate/</guid>
      <description>&lt;p&gt;Wow, arithmetic is potentially inappropriate!&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/text-generation-playground?mode=text&amp;amp;modelId=amazon.titan-text-lite-v1&#34;&gt;https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/text-generation-playground?mode=text&amp;amp;modelId=amazon.titan-text-lite-v1&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;img loading=&#34;lazy&#34; src=&#34;https://files.s-anand.net/images/2024-11-09-arithmetic-is-inappropriate-linkedin.jpg&#34;&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;https://www.linkedin.com/feed/update/urn%3Ali%3Ashare%3A7260990576433520640&#34;&gt;LinkedIn&lt;/a&gt;&lt;/p&gt;
</description>
    </item>
    <item>
      <title>The three Rs</title>
      <link>https://www.s-anand.net/blog/the-three-rs/</link>
      <pubDate>Mon, 23 Jan 2012 18:13:08 +0000</pubDate>
      <guid>https://www.s-anand.net/blog/the-three-rs/</guid>
      <description>&lt;p&gt;Reading, wRiting and aRithmetic are the 3 &amp;lsquo;R&amp;rsquo;s that are taught at school. I was thinking about their relevance today.&lt;/p&gt;
&lt;p&gt;Reading continues to be relevant. The volume of information available today is more than before. So you need to read faster AND smarter. (If there was one good thing that came out of my IIM coaching classes, it was the ability to read fast, and making it subconscious.)&lt;/p&gt;
&lt;p&gt;But I wouldn&amp;rsquo;t say the same of writing. In the last 10 years, I have typed several hundred more pages than I&amp;rsquo;ve written. So have all my friends.&lt;/p&gt;
&lt;p&gt;Yesterday, I was at a bank with a relationship manager as he was taking notes in paper and pen. I do the same on occassion. I looked at his notes later. I could not understand a single word. &amp;ldquo;Don&amp;rsquo;t worry, sir, I can read it. I&amp;rsquo;ll type it out and mail you,&amp;rdquo; he said. And he did.&lt;/p&gt;
&lt;p&gt;Writing seems to have become a device for personal memory, not communication. He&amp;rsquo;s faster at writing than typing, perhaps. Or note taking is more convenient on paper. But for communication, he still prefers a typed format. So do I, and most other people.&lt;/p&gt;
&lt;p&gt;Perhaps writing will fade. Perhaps not. I don&amp;rsquo;t know. But what I do know is that typing has become more important than writing. Yet, writing is taught more at school than typing.&lt;/p&gt;
&lt;p&gt;(A broader aspect of writing, though, is expressing oneself. That will remain important, of course.)&lt;/p&gt;
&lt;p&gt;The third R is aRithmetic. When I was 12, I could multiply four-digit numbers in my head reasonably well. I could recite 50 digits of Pi. I could do long division. Today, I can&amp;rsquo;t. Nor can my friends. Nor have we needed to. A good feel for the numbers has helped, but not the actual mechanics of the calculations.&lt;/p&gt;
&lt;p&gt;We had an undergraduate course in statistics that taught us how to solve a linear regression problem. That skill went completely unused. I&amp;rsquo;ve never since used regression without a computer. We had a graduate course in statistics that taught us how to INTERPRET the results of a linear regression. That was worth it&amp;rsquo;s weight in gold.&lt;/p&gt;
&lt;p&gt;This is not a critique of the three Rs. Rather, an attempt to re-interpret them. It&amp;rsquo;s about comprehension, expression and computation. Two decades ago, it was reading, writing and arithmetic. Today, it&amp;rsquo;s reading, typing and computing.&lt;/p&gt;
&lt;p&gt;Computers will grow more powerful. It may be worth planning for it. Teaching the ability to use them can go a long way. A tool like Excel for general purpose computing gives incredible power in the hands of people. It&amp;rsquo;s worth training children for that.&lt;/p&gt;
&lt;p&gt;If I oversimplified, I&amp;rsquo;d say children must learn typing and Excel.&lt;/p&gt;
&lt;p&gt;Over the next few years, this is something I plan to work on. Making sure schools and parents do this. Any suggestions or leads you may have are welcome!&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id=&#34;comments&#34;&gt;Comments&lt;/h2&gt;
&lt;!-- wp-comments-start --&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;a href=&#34;http://arunrocks.com/&#34;&gt;Arun&lt;/a&gt;&lt;/strong&gt; &lt;em&gt;7 Feb 2012 12:05 pm&lt;/em&gt;:
I find the 3 Rs to be redundant but still useful for cognition. I find any computer input device to be limiting because I cannot doodle or sketch if I need to while mapping an idea. Paper offers freedom to graphically detail a problem in ways that cannot be replicated easily or effortlessly on a computer (or a tablet for that matter). Somehow whiteboards the best way to communicate to a team than Powerpoint.
Agree about aRithmetic. We almost never find ourselves without a computing device . But should we be completely reliant on them? What if you are marooned in an island? :D
Today, I would simply add a fourth R - &amp;lsquo;Recursion&amp;rsquo;. Implying the art of programming, the lack of which has become a serious handicap.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Somnath&lt;/strong&gt; &lt;em&gt;23 Jan 2012 7:04 pm&lt;/em&gt;:
You don&amp;rsquo;t need to teach typing I guess - kids these days have computer at home if not at private schools. My 1.5 year old has seen me type and has taken to imitate that. Over time they will learn typing out of necessity. Excel I agree you need to teach as it might not be seen as necessary - not Word or Powerpoint though as they will pick it up out of necessity due to school work.
Having said that I still carry a simple ball-point pen and a paper whenever I get in a meeting or on my desk to take notes. No technology can get in that as I see it as obstructive.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;a href=&#34;http://www.verge2.com&#34;&gt;Ken Jones&lt;/a&gt;&lt;/strong&gt; &lt;em&gt;22 Feb 2012 6:59 pm&lt;/em&gt;:
Anand, I stumbled across your site 4 years ago and have always loved it.
I agree that Excel is a wonderful tool to teach. It has the beauty of being as simple or and fantastically intricate and complicated tool you can make it. And these days you can link it to all the xml you want.
I have always thought that logic - especially as it applies to programming - is very useful. However, I feel that there need to be more visual tools to assist in its education. Kids are able to follow nested loops and if/then stateemnts if they have an easy way to navigate the logic initially. I became very interested in the way Montessori teaches mathematics at an early age. But I have yet to see effective tools like that for logic education at a young age ( 4-10)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Judy H&lt;/strong&gt; &lt;em&gt;23 Feb 2012 6:55 am&lt;/em&gt;:
I have noticed for a long time that my handwriting has gone to hell ever since I&amp;rsquo;ve had a computer at my desk. I had nice handwriting, and now it looks awful and feels cumbersome as well. I have heard that school districts have stopped teaching cursive writing to children. Imagine! Not that I disagree, how much really free hand writing does anyone do anymore? Maybe this will free up time to teach them things they really need, as some noted already: logic, analytical thinking, problem solving, and common sense. My background is in math and statistics. I spent many years working with people in other sciences who had a tough time working through a problem, or setting up an experiment, in a logical and thoughtful way &amp;ndash; planning and executing, learning from the result, and setting up the next iteration. These types of skills, along with communications skills (getting your point across as well as listening), are critical for success in life. I don&amp;rsquo;t think Excel has entirely taken away the need for learning arithmetic, but I would like to see kids get an answer to a problem and know whether they have gotten the right answer (so some time of quick estimation method is needed). I emphasized it with my kids &amp;ndash; you can easily use a calculator, but how do you know you got the right answer? Does the answer you got make sense? To round out the basic public school education, I would include lots of other topics, but mostly to expose kids to what is out there &amp;ndash; literature, geography, philosophy, anthropology &amp;ndash; to make them more curious to explore on their own. Every thing is at your fingertips these days, I love it! Why are we still forcing kids to read books they hate? I would rather let them gravitate toward what thrills them &amp;ndash; Lord of the Rings instead of Huck Finn, or Tolstoy instead of Shakespeare, whatever turns them on.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Rajesh Bhura&lt;/strong&gt; &lt;em&gt;14 Mar 2012 9:35 am&lt;/em&gt;:
Anand, what do you think about the utility of tools like abacus for teaching aRithmetic to children? Spandana who is 9 yrs old has been going to this UCMAS class for 3-4 yrs and i am still to find true value in this. i know i am deviating from the main discussion but wanted to know your and the views of others pls.
&lt;a href=&#34;http://www.ucmasindia.com/mental&#34;&gt;http://www.ucmasindia.com/mental&lt;/a&gt;_arithmetic/aboutusnew2.html&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;a href=&#34;http://tildekarthik.blogspot.com&#34;&gt;Karthik A&lt;/a&gt;&lt;/strong&gt; &lt;em&gt;12 Feb 2012 3:37 am&lt;/em&gt;:
I feel that logic (aRgument) helps increase flow for children and sometimes, like sleeping a problem over, helps in generating leaps of logic (intuition or conceptualization). We need to teach these in place of writing.
Long back I remember my grandfather telling me that they used to have logic as a subject!&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Ravi&lt;/strong&gt; &lt;em&gt;10 Apr 2012 10:40 pm&lt;/em&gt;:
&lt;a href=&#34;http://www.themodernword.com/eco/eco&#34;&gt;http://www.themodernword.com/eco/eco&lt;/a&gt;_future_of_book.html
This is from Umberto Eco, one of the foremost and important intellectual of our times. This is his area &amp;ldquo;Semiotics&amp;rdquo; - the science of signs.
Somehow I remembered this discussion and thought it would make sense to post this here.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Ravi&lt;/strong&gt; &lt;em&gt;20 Feb 2012 9:44 pm&lt;/em&gt;:
The three R&amp;rsquo;s are going to be relevant as long as civilization exists - their forms and formats will undergo constant shifts and changes. All these three R&amp;rsquo;s are at some level only abstractions. If you think about it, reading was always a tool of internalization of concepts and communicating and creating within, writing was for externalization and communication without, and arithmetic was just a name given to the abstract name for the reliazation of the natural phenomena through a number line. By writing, if you meant translating your thoughts on a paper using a pen/pencil - yes that is receding; but the act of translating your thoughts onto something has not changed. In fact, if a medeival saint who perhaps wrote on &amp;ldquo;tala&amp;rdquo; leaves using a stylus magically appeared in 1950&amp;rsquo;s, he would be flabbergasted by the meaning &amp;ldquo;writing&amp;rdquo; had acquired; same surprise that our grandfathers would express watching our kids type on computer screens and tablets. Arithmetic, however if narrowed down to doing math in the mind has definitely as an academic discipline has taken a violent beating at the hands of xcel, the ubiquitious calculator; but the concept of numbers has not. Neither will till the civilization exists.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;deepan&lt;/strong&gt; &lt;em&gt;24 Jan 2012 8:12 am&lt;/em&gt;:
Programming is still a subject that scares many. Misguided Focus on syntax rather than on logic and problem solving ,makes the learner diffident .Excel offers an easy way to dabble ,solve problems and experience sheer joy of working smart.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Shankar V&lt;/strong&gt; &lt;em&gt;24 Jan 2012 9:21 am&lt;/em&gt;:
I agree with Somnath. Note taking is still paper and pen in my case. I tried using the evernotes, the onenotes and the sticky notes. I simply lose my way there. With a notebook, it is all chronologically sequenced and I know what I am looking for the moment I open the pages.
But writing skills other than personal note taking is almost dead. And we still have schools teaching &amp;ldquo;cursive writing&amp;rdquo; skills to kids. :)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Rajesh B&lt;/strong&gt; &lt;em&gt;26 Jan 2012 6:04 am&lt;/em&gt;:
Good points. A redefinition of the 3 R&amp;rsquo;s is definitely called for. But, I would still add typing and excel as auxiliary skill sets, and spend lots of time on the basics. Excel, typing are more &amp;ldquo;skill sets&amp;rdquo; to have. At very young ages, it is very important to build the mind to be able to process stuff, rather than be able to &amp;ldquo;know&amp;rdquo; stuff. Kind of like the difference between academic education and vocational education. No offence to excel, but it clearly falls under &amp;ldquo;skills&amp;rdquo; category. As in, if we can make our kids more receptive and intelligent, they can figure out excel without much difficulty.
Writing (in the communication aspect, not the putting pen to paper aspect), Reading and arithmetic all play a role in shaping the mind. Comprehension, articulation and analysis (not necessarily just computation) are important bits in piecing together a keen mind. I would look at all of schooling as the phase where the mind is made better. And not really worry about pure &amp;ldquo;skill sets&amp;rdquo;
In computer terminology, till age 16 or 18, work on the RAM. The apps can come later. If your RAM is cranked up really well, loading apps is very easy. And the hard-disk generally is sound enough for the basics as well.
Most of the students that I interact with (all in the 21-30 age group) have good apps, hajaar heavy hard-drive, but are running on a 128 MB RAM.
I would want entire schooling to focus on improving processing ability and stamina. Everything else can come later.
We can (and probably should) de-emphasize writing, and supplement it with typing. We are essentially working on articulation. As far as arithmetic goes, we should de-emphasize computation and increase the role of context and analysis. Include logical reasoning type questions in the pedagogy in order to combine analysis and comprehension skills. We should definitely expand the role of reading.
But I am a little old-fashioned so would want to keep original frameworks. My biggest fear is that we introduce an app in the hope that it will improve processing power and somewhere down the line it becomes a task in itself and loses all value.
This is what has happened to our ability of multiplying two 2-digit numbers mentally. Originally, it was a way to teach kids to have an intuition about size. Now, it is a cr*p vedic maths module with no basis. Pardon my language, Anand.
I do not know if I have communicated clearly here. When you (Anand) were asked to do 47 * 74, it might have been to tell you that it is somewhere close to 50 * 70. And to help you to process and remember at the same time. Now, it is a template and there is no-one who discusses how this is close to 50 * 70.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;a href=&#34;http://www.amlan.in&#34;&gt;Amlan&lt;/a&gt;&lt;/strong&gt; &lt;em&gt;17 Feb 2012 3:12 pm&lt;/em&gt;:
Products like the Livescribe smartpen will merge the worlds of writing and computing.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Ravi&lt;/strong&gt; &lt;em&gt;10 Apr 2012 10:44 pm&lt;/em&gt;:
I hasten to add that this was a lecture in a 1994 symposium when Steve Jobs was still disillusioned about Apple and was making movies and not dreaming his &amp;ldquo;i&amp;rdquo; devices :-) yet the discussion is useful and the cross-references to historical and intellectual motifs are invaluable.&lt;/li&gt;
&lt;/ul&gt;
&lt;!-- wp-comments-end --&gt;
</description>
    </item>
    <item>
      <title>It is possible to divide by 3</title>
      <link>https://www.s-anand.net/blog/it-is-possible-to-divide-by-3/</link>
      <pubDate>Wed, 14 Aug 2002 12:00:00 +0000</pubDate>
      <guid>https://www.s-anand.net/blog/it-is-possible-to-divide-by-3/</guid>
      <description>&lt;p&gt;Conway and Doyle prove that it is &lt;a href=&#34;http://hilbert.dartmouth.edu/~doyle/docs/three/three/three.html&#34;&gt;possible to divide by 3&lt;/a&gt;. The paper, which is distributed under the GPL, unfortunately comes without a warranty. (via &lt;a href=&#34;http://gimbo.org.uk/&#34;&gt;Gimbo&lt;/a&gt;)&lt;/p&gt;
</description>
    </item>
  </channel>
</rss>
