heuristics on S Anand

Testing Pólya heuristics on AI Math

Fri, 27 Mar 2026 13:20:25 +0800

Terence Tao said, “We haven’t done many experiments … large-scale studies where we take a thousand problems and just test them.”

So I told Claude: You know my style. Suggest some innovative experiments I could run.

The first suggestion was cool! The Polya Audit. Polya’s How to Solve It lists 20 heuristics (work backwards, induction, analogy, etc.). Mathematicians treat these as wisdom. Nobody has ever measured which ones actually work, and on what problem types.

So I prompted Copilot running Claude Sonnet 4.6 to run the LeanDojo Benchmark through an LLM n times, with different Polya heuristic system prompts and compare success rates.

Not-surprisingly different heuristics help different problems.

Almost every heuristic helps Prealgebra - except “Start from the desired answer and reason step by step back toward the given information.”.
Almost no heuristics helps Number Theory - except “Focus on the largest, smallest, or boundary element. Extremal elements often have special properties.”.
Geometry has an enormous swing. “First strip away complexity and solve an easier version. Observe the pattern, then generalize” helps a lot. But “Find a quantity that can be counted in two different ways. Set up both expressions and equate them” hurts a lot.

The impact of each heuristic is also quite varied.

The most reliable heuristic is segmentation: “Identify the key condition that splits the problem. List all possible cases exhaustively. Handle each with a complete argument.”
The worst heuristic on average is pattern recognition: “Compute several specific instances. Tabulate results. Identify a pattern. State the conjecture. Then prove or use it.” Induction and pigeonhole do pretty bad, too.

Also not-surprisingly, different models respond differently to the same heuristic.

GPT-5.4-nano: Heuristics disurpt it. Its built-in problem-solving strategy is already good. Heuristic just make things worse, almost always.
Gemini 2.5 Flash Lite: More coachable. Like a student who benefits from advice: it gains up to 6 percentage points from the right heuristic.
Claude Haiku: Nearly immune. It seems to just ignore the heuristic. Its performance barely moves regardless of what you tell it.

The same heuristic on the same problem affects models quite differently, too. For example “Introduce Auxiliary Elements” hurts GPT -25% but helps Claude +14%!

So yes, different heuristics work for different problems, and different models respond differently to the same heuristic.

But finally, at least for LLMs, we can measure. We can find out which heuristics work for which problems, and which heuristics get varied responses vs which ones are more universally helpful / harmful. And maybe teach humans.

Or maybe not.

Implicit information

Thu, 03 Jan 2008 12:00:00 +0000

From what I’ve seen, puzzles and exam questions share two un-real-worldly characteristics. Firstly, you are guaranteed that a solution exists. Secondly, you are given that all the information provided to you is relevant. (Well, not always. Some case studies I’ve seen have had their share of contrived irrelevance. But that’s often what it is, I think. People fill in the relevant stuff, and then try and distract by adding irrelevant material in the hope of making it more real-world-like. But that’s just a guess).

These are very powerful constraints. I know of nothing that has given me as much confidence in solving puzzles as the assurance that a solution exists (and that someone thinks me capable of getting it).

But it’s more than just a confidence builder. The guarantee that a solution (and invariably it’s a unique) is a very powerful one. An extreme case is an objective type question, which explicitly provides three guarantees:

There is a solution
There is only ONE solution
It is among the choices listed below

(Some papers try and take away the first guarantee by having an (E) None of the above category. But that’s still leaving behind the other two more powerful guarantees.)

Marking answers randomly, or marking (A) for every question would still get you 25% in an exam with 4 choices. (Marking (C) would prove just as good, unless you had a kind professor like this.) That’s better than any real-world scenario I’ve seen. (Real-world strategies aren’t much better, though.)

Using guarantee 2, you can eliminate choices easily. If (A) and (B) do not satisfy some property of the solution, they CANNOT be the answer. There’s only one solution, and these are not it.

Using guarantee 3, you can pick the last remaining choice wihout having to check it. The solution is definitely among the choices listed. So you don’t need to solve an objective type question. You just need to pick the right answer – which is completely different.

The principle applies even outside of objective type questions, especially in mathematically-oriented problems, or puzzles. And you can solve it by trial and error. For example, try this one from Martin Gardner’s Mathematical Magic Show:

Two brothers own n sheep, each of which is sold for n dollars. Thus they have n2 dollars in all. This is in the form of 10 dollar notes and 1 dollar coins, the number of 1 dollar coins being less than 10 dollars. The elder brother divides the money as follows: he takes a note for himself, gives one to his younger brother, takes a note for himself and so on. At the end, the younger brother complains that the elder took the first note as well as the last. So the elder gives the younger all the one dollar coins. The younger brother complains that he still has more. So the elder brother writes the younger a cheque to equalize their share. What was the cheque for?

Now, this is a weird problem. Think about it. You’re told almost NOTHING. And you have to guess what the amount is. (Note: you don’t have to guess what ’n’ is. That’s impossible.)

Here’s how I solved the problem. I said, let me find even one case where the elder brother gets the first and last note. Let’s see what the answer is. Whatever the answer is for that case, it has to be the answer for all other cases – because otherwise, the problem does not have a unique solution.

So I tried n=1. n=2. n=3. For n=4, the amount is 16. That’s 1 $10 note and 6 $1 coins. The elder brother would get the first and the last $10 note. The younger would get $6. So the elder would have $4 more than the younger, and would write out a cheque for $2. (It’s amazing how many people get as far as the $4, but forget to divide by two.)

You can try if for any other value that has an odd number of $10 notes. It has to be for n ending with 4 or 6. That means n2 ends in 6, and the cheque has to be for $2.

Notice that you didn’t need number theory to get the answer. The assurance that there is a unique answer is enough.

There’s another kind of implicit information usually available: the amount of information there is. For example, take the following question:

Which city has a higher population: San Antonio or San Diego?

Children in the US apparantly had difficulty answering it. Children in Germany had less trouble. The reason? The German kids had heard of San Diego, but not San Antonio. They figured the one they’d heard of was more likely bigger. Knowing less may be better.

It’s the same principle you use to check spellings. Run a Google search on two spellings. The one that returns a higher number of results is the correct spelling. (Of course, Google has a spelling correction mechanism that works well, but I use it for Tamil words. I can never tell if I should use ர or ற.)

Of course, the fundamental assumption here is: MORE INFORMATION = MORE CORRECT, which is not always the case. But the point I’m driving to is this:

You’re always given additional information. Even if you’re not given any information, that’s informative.

Comments

Sathyaraj 3 Jan 2008 12:00 pm: this article was “fucking” good..i was reading a book called fooled by randomness and i just could relate to the way you and the author went about writing this.. can you write more about monte carlo simulation too?? and how it is used in finance…
Vasanth 3 Jan 2008 12:00 pm: Very interesting post! New insight for me into puzzles. I had always realized, but never analyzed, that I never give up when it comes to puzzles and brain teasers. On the contrary I give up relatively easily with some technical or work related problem the minute it appears probable that a (reasonable) solution does not exist.
S Anand 3 Jan 2008 12:00 pm: @Sathyaraj: Thanks! The sequel to “Fooled by Randomness” is pretty good to: “The Black Swan”. @Vasanth: Another big problem, if you ask me, is that the problem is unbounded. In puzzles, you have a test case to determine whether you’ve solved it or not. In life, you aren’t often sure.
Jayanth Sankar 3 Jan 2008 12:00 pm: Hey I like your posts. Pretty good. The question about the money between the two brothers.. I have a doubt here.. you had mentioned number of 1 dollar notes should be less than 10 dollar notes, so then 16 wont work right ? 1 - $ 10 and 6 - $1… what am I missing ?

Knowing less is better

Wed, 17 Jan 2007 12:00:00 +0000

Malcolm Gladwell argues that knowing less can be an advantage. This is based on a study in which kids in the US were asked which was a bigger city: San Antonio or San Diego. Many didn’t know. Kids in Germany were asked the same. Most knew: San Diego was bigger. Why? Because they’d heard of San Diego, but not of San Antonio.

P.S: A comment mentions that the actual difference in population between these cities is only 2%. So maybe the US kids were right to be unsure…

Comments

texan 29 Jan 2007 4:20 am: Please read the actual pschologist’s blog at http://www.dangoldstein.com/dsn/archives/2006/03/malcolm_gladwel_1.html#note to see the distinction between “city limits” & “metro area”. i or any outsider ( still a US person ) would associate san diego(San Antonio) with its metro area than ‘city limits’. I think the american kids were influenced by the popular saying ‘Everything is big in Texas !!!’ i live in Texas BTW

Programming theorems

Wed, 08 Nov 2006 12:00:00 +0000

Programming theorems.

The likelihood of Perl being involved in a system is directly proportional to the length of time the system has been in maintenance.

Every 5 minutes you spend writing code in a new language is more useful than 5 hours reading blog posts about how great the language is.

Think twice before presuming that CSV is a nice little easy file format. (see Leon)

Errors in multicriteria decision making

Mon, 09 Oct 2006 12:00:00 +0000

I talked about my approach for multicriteria decision-making, and mentioned that it was fundamentally flawed. Here’s why.

The charts above compared two industries. The bigger the area, the more favourable the industry. The underlying assumptions being:

The criteria are comparable. (Points at the same level are of comparable importance. Twice as large is twice as important.)
All (and only) relevant criteria have been included.

In this particular example, I know for a fact that both these assumptions are invalid. And in every case I used this methodology, the assumptions fail.

You won’t draw the criteria to scale. We used revenues and growth as two parameters, and marked each industry as high, medium or low. The scale for revenue was Rs 100 cr, Rs 500 cr and over Rs 500 cr. The scale for growth was <5%, 5-10%, >10%. We picked this scale in order to fit the range well on these graphs. Not because Rs 100 cr of revenue was worth about the same as 5% of growth. And yet, that’s the implicit trade-off this graph is asking us to make.

We also had very qualitative criteria, like “Capability” (KSF), and they were compared head-on with growth and revenue. Using qualitiative criteria is not a bad thing. But when the visual makes you trade-off capability against Rs 500 cr of revenue, I feel queasy.

You will miss important criteria. Usually, the process for identifying criteria is bad. “Think of every criteria you can” was our standard approach. In this instance, in our first iteration, we had a dozen parameters. We showed it to the client. They said, “Look, our Chairman likes these industries a lot. He doesn’t like that bunch. We’re much more likely to focus on the ones he likes.” And that’s absolutely important! We ended up adding a “Passion / vision” based on the fit with the company’s existing businesses, and that proved the deciding factor.

Another time, I built an entire model on which project to outsource based on 10 parameters. (It was everything I could think of at the time.) The one that I missed was, “When is the project starting?”. It turned out that this was the most important criteria. In fact, it was the only important criterion. If I’d simply said, all projects starting after 1-June-2006 can be outsourced, I would’ve been 90% right.

You’ll keep the irrelevant parameters. This is the worst of all. Even after we learnt which the important criteria were, we didn’t throw away the useless parameters. We never throw away hours of work, even if it’s useless. So the model keeps bloating, and the irrelevant criteria influenced the shape of the graph more than the relevant ones.

Another problem is that this methodology cannot answer questions concisely.

“Why did you knock off Industry X?”

“Oh, because on a cumulative score against revenue, growth, lifecycle, capability, passion and 10 others, it scores less than 45 points.”

A good answer should be short. For GE, it would be “You’ll never be number one.” For HP, once, it would’ve been “It’s not where we can excel technically.” For Warren Buffet, it may be “I don’t understand the business.”

After these experiences, and based on hindsight, I’ve come to believe the following about MCDM (multi-criteria decision making):

Remove irrelevant criteria. Usually a few criteria make the decision. The rest don’t matter.
Filtering works. You don’t optimise in MCDM. You’re trying to do well against criteria that are often not comparable. You’re better off if you filter out unacceptable levels of criteria, and treat what’s left as acceptable.
Use a decision tree. Don’t compare options. This is the basis of fast and frugal heuristics. It keeps focus on the important criteria, and makes the process easy to implement / explain.

I’ll let you read up on fast and frugal heuristics. I’m convinced it’s the best way to make decisions based on multiple criteria in the scenarios I’ve worked on.

Comments

Shirali 10 Oct 2006 11:49 am: How about BCG Matrix?
ND 10 Oct 2006 3:25 pm: Ask client (decison makers or those close to them) for the important criteria after the first cut and prune the list Remove irrelevant criteria and possibly assign weights! I’m uncomfortable of areas under a spider chart. Prefer to use “moon phases” that are backed by some data analysis. Comparison is essential for an “either- or” decision (Should I outsource or not?). Decision trees work when its an “open mind” situation (What kind of applications should I outsiurce?). Endnote - BCG matrix appears a star… it’s a dog’s world after all ;-).
Sandeep 11 Oct 2006 11:54 am: Anand, I buy your argument about comparing qualitative and quantitaive measures. I had to do a similar excersise for one of my projects. I used a weighted average model.What I did was 1. Identified all the relevant factors to grade each industry, as agreed upon by the necessary stakeholders 2. Assigned weights to each factor…again with stakeholder agreement 3. Graded/ranked all the industries by each factor and totalled up a weighted average number. So finally I had one number per industry, to do my comparison. Isn’t this method better as compared to representing on a wheel chart. Your thoughts….??
Jawahar 9 Oct 2006 12:00 pm: Hai anand, can you mail me some links on multi criteria decision making and which method is best in multi criteria decision making ? though you have told weighting is wrong,its hard to see how not to use weighting in decision models like AHP so can you throw some light ?

How to pick a course

Thu, 29 Jun 2006 12:00:00 +0000

In his article on The Power of the Marginal, Paul Graham suggests (among other things) a way of picking courses at college.

One way to tell whether a field has consistent standards is the overlap between the leading practitioners and the people who teach the subject in universities. At one end of the scale you have fields like math and physics, where nearly all the teachers are among the best practitioners. In the middle are medicine, law, history, architecture, and computer science, where many are. At the bottom are business, literature, and the visual arts, where there’s almost no overlap between the teachers and the leading practitioners. It’s this end that gives rise to phrases like “those who can’t do, teach.”

Incidentally, this scale might be helpful in deciding what to study in college. When I was in college the rule seemed to be that you should study whatever you were most interested in. But in retrospect you’re probably better off studying something moderately interesting with someone who’s good at it than something very interesting with someone who isn’t. You often hear people say that you shouldn’t major in business in college, but this is actually an instance of a more general rule: don’t learn things from teachers who are bad at them.