statistics on S Anand

Mon, 10 Nov 2025 00:00:00 +0000

I always wondered why old movies are rated so high on IMDb.

For example, 12 Angry Men (1954) with just ~900K votes ranks about as high as Inception (2010) with ~2M votes. Few people I know have seen 12 Angry Men. So where does this high rating come from?

My theories were:

Old movies really are that good.
IMDb’s algorithm is biased towards old movies.
People remember older movies fondly.

Actually, it’s none of these. It’s selection bias.

Few people watch a 1950s black & white drama: cinephiles, film students, etc. They love it and give it 9s and 10s.

Everyone watched Inception. The casual majority thinks it’s fine, not life changing. Maybe a 7 or 8.

This creates a paradox: obscurity protects ratings while popularity is its own punishment. Only “devotees” watch obscure movies - leading to better ratings than widely seen movies.

PS: This data analysis and story were authored by Claude Code. That includes the statistical significance validation.

Story: https://sanand0.github.io/datastories/imdb-democracy-penalty/index.html Prompt: http://github.com/sanand0/datastories/tree/main/imdb-democracy-penalty Code: https://github.com/sanand0/imdbscrape/pull/1

Correlating subjects

Sun, 12 Feb 2012 04:08:02 +0000

A question from Dorai get me thinking: does being good at maths help in programming?

I don’t have a personal view. But since Reportbee has data on the Class 12 examination results for the last three years, we thought we could do a bit of analysis.

Here’s the correlation of the scores of various subjects with Computer Science.

Correlation	Subject
0.79	CHEMISTRY
0.79	PHYSICS
0.75	ENGLISH
0.75	MATHEMATICS
0.72	LANGUAGE
0.67	BIOLOGY
0.66	ECONOMICS
0.66	COMMERCE
0.65	ACCOUNTANCY
0.56	HISTORY
0.52	GEOGRAPHY

It almost breaks neatly into four groups.

Physics & Chemistry, both of which have a correlation of 0.79, and clearly are the most correlated with Computer Science
Maths, English & Language, which have a correlation of 0.72 – 0.75
Biology, Economics, Commerce and Accountancy, which hover at around 0.66
History & Geography, which are 0.52 – 0.56

The results in 2010 are almost exactly the same.

Correlation	Subject
0.78	PHYSICS
0.78	CHEMISTRY
0.75	ENGLISH
0.75	MATHEMATICS
0.73	LANGUAGE
0.67	ACCOUNTANCY
0.65	ECONOMICS
0.65	COMMERCE
0.64	BIOLOGY
0.60	GEOGRAPHY
0.55	HISTORY

I’m not sure what it is that leads to this kind of correlation. In fact, the full correlation between every pair of subjects (for 2011) is below:

What inferences would you draw from this?

And what do you think is the reason for this?

Comments

Arun Ravindran 12 Feb 2012 5:53 am: My inference is purely anecdotal but might be helpful in explaining this data. I had chosen Computer Science for my +2 in 1998. It was taught in C++ and mostly involved memorising operations on Data Structures such as Lists, Stacks and Queues. The C++ standard library had to be memorized including I/O, string and file functions. The exams were basically a test of memory rather than attacking a new problem space mathematically. Guess which are the other subjects which involve memorising a huge set of symbolic facts? – Chemistry and to a certain extend, Physics. I believe the data is more revealing of our Computer Science pedagogical and evaluation methods than the subject itself.
S Anand 12 Feb 2012 6:46 am: Good point. It’s quite debatable whether marks in computer science are indicative of programming ability.
Anamika 12 Feb 2012 10:21 am: Netruvarai neram poga villaiyae, unadhu arugae neram podhavillaiyae…!
Sathya 16 Feb 2012 12:56 pm: In my experience, I’ve come across rock star programmers who have sound grasp of mathematics. But i dont have statistics to prove them. As we all know, programming (esp functional) is heavily influenced by mathematics. Coming to inferences, I would take the dataset with a pinch of salt. Is it diverse enough to be statistically significant ? I agree with most of the comments on relating “mugging up programs” to being good at Chemistry in particular.
Sathyaraj 12 Feb 2012 7:44 pm: My 2 cents based on experience. People who take the computer science group generally have to always take one language and English subject apart from taking Physics, Chemistry, Maths. They would not be able to take classes in accountancy, economics, history, etc. Some of the students at the top of the class would have realized that in order to differentiate in terms of coming first in class/school, one would need to excel in language and english. It is given that one needs to get excellent scores in physics, chemistry and maths to get an overall good percentage. Also the fact, that scoring a big total would enable them to get admission in colleges in like BITS pilani which looks at overall score. Students who take the biology or accountancy group have no such pressure for them to excel in English and language subjects. Hope it makes sense. Also, if you can do a correlation between computer science students and the language that they take, I believe you will find that the majority of students would have chosen french rather than Tamil or English.
Shankar V 13 Feb 2012 6:23 am: Anand Correlation is a misleading statistic if the dependent and independent variables are not “really” related that way. There could be compounding effects within the independent variables leading to wrong correlation coefficients. There are ways to detect and cleanse compounding - design of experiments (in statistics) deals in depth on this. It is quite possible that there could be some relationship in how the computer science and math exams are scheduled and the scores in these subjects. If they are too close to each other, the student may have had lesser time to prepare for the math exam. For example, if Physics and Math were scheduled one after the other, and Computer Science test is after the Math test, the student may have prepared better for Physics, not have had enough time for Math and then recovered to do well in Computer Science. This can be a pattern in the entire class as it is normal for kids to focus more on Physics (the dreaded subject!) v/s Math. The statistic may not reveal ability or natural alignment of the subjects.
Evgeni 11 Dec 2012 3:37 am: I’d rearrange subjects such that they’re more clustered together by correlation, like in a TreeMap view. That way it’s easier to see the relationships.
amanjot kaur 10 Sep 2015 12:28 pm: i need correlation of commerce with language

Birthday matters

Sun, 15 May 2011 20:33:23 +0000

Does it matter which month you’re born in?

Based on the results of the 20 lakh students taking the Class XII exams at Tamil Nadu over the last 3 years (via Reportbee), it appears that the month you were born in can make a difference of as much as 120 marks out of 1,200 – or 10%!

Most students who took the Class XII exams in 2011 were born between March 1991 and June 1992. The average marks of each student (out of 1200) is shown in the graph below.

Students born in June 1991 scored the lowest – around 720/1200. This suddenly shoots up in July, then in August, and the students born in September score as much as 840/1200 on average. From there on, it’s downhill.

This result is consistent across years. In 2009 and 2010, you see a similar pattern.

Why could this be?

Malcolm Gladwell’s book Outliers offers a clue.

Outliers opens, for example, by examining why a hugely disproportionate number of professional hockey and soccer players are born in January, February and March.

The answer turns out to be completely unrelated to numerology or astrology.

It’s simply that in Canada the eligibility cutoff for age-class hockey is January 1. A boy who turns ten on January 2, then, could be playing alongside someone who doesn’t turn ten until the end of the year—and at that age, in preadolescence, a twelve-month gap in age represents an enormous difference in physical maturity.

In Tamil Nadu, students must be 5 years old before entering Class 1. Schools open mid-June. So students born in June 1994 would barely make it in June 1999 – making them the youngest students in the class. July and August students would be missed – but since many schools implement this policy leniently, they sometimes make it in as well. September borns are often consistently the eldest students in a class.

This pattern reflected in the marks. The eldest – the September 1993 borns – score the highest. The next eldest, the October 1993 borns, score a bit less. And so on. (There are older students who take the exam – the ones born before September 1993 – but many of these are failed students from the previous year, introducing a bias in the results.)

Perhaps this initial advantage that the elder students have over their classmates continues through the years? Whatever the reason, it’s clear that if your child is born in September, he or she already has a 100 mark advantage!

Comments

abhijeet 16 May 2011 8:14 am: “Whatever the reason, it’s clear that if your child is born in September, he or she already has a 100 mark advantage!” But at the cost of being older and “losing” 8-9 months!
Shankar V 16 May 2011 10:00 am: Nice…!! Thank God I was born in July! If not on the crest, I at least made it into the upward curve! ;)
S Anand 16 May 2011 10:21 am: @abhijeet: That reminds me of the “Life is a race” dialogue in 3 Idiots :-) Well, not sure how bad being a bit older is. I personally don’t mind, but maybe others do?
Shailesh 16 May 2011 10:33 am: Oh…now I know why I was getting good marks…. :-) I am born in September…and my headmaster changed my birthdate (to Jul month) in school register to not waste my 8-9 months…..
ravi atluri 16 May 2011 10:47 am: They made me study LKG “TWICE”, because I was born in September and didn’t have enough time to be 3! and I have been an anomaly to the september-born-high-performers trend. If the marks are related to the mugging capacity, which if related to age would explain the graphs only for the schooling during. Would like to see, how would these change in college and bachelors :D One more doubt, I have is, many parents have the dob’s of after-aug-born’s to sometime between may to june. If that’s the case, the months on the graph have to be offset ;)
S Anand 16 May 2011 12:35 pm: @Shailesh – if you got into school early, then statistically, you had a DISadvantage. So guess you were MUCH brighter than your classmates! @Ravi – true. No idea how to correct for the real date of birth, though…
k 17 May 2011 2:53 am: this is similar to the paper on Israeli schools where an externally imposed rule created a disproportionate class size, which helps in identifying the effect of class size on performance. date of birth can help in identifying the effect on performance then.
Vasant 17 May 2011 6:14 am: Ha! Now I know why I scored so less in the boards. It’s when I was born! :-) Jokes aside, brilliant analysis.
Sathya 17 May 2011 6:43 am: Good One ! If you still have access to data, would it be possible to plot the same graph against their “numerology numbers” ? ie., 28th born would be 2+8 = 10 => 1+0 = 1. I used to wonder that astrology ( perhaps numerology ) could provide certain insights and that it astro/numero predictions should be verified using statistics. ( Pls note that I do not suggest astro determines a person’s fate. But perhaps it can predict events with some accuracy. How a person reacts to those events are not in the realm of astrology). I would love to do this if I had access to the data. In case you find it worthwhile, it would be interesting to see if numero number matters at all !
Swetha 19 May 2011 9:34 am: M a September born and a gold medalist . (*Grins*)
Sumit Dhar 21 May 2011 3:36 am: And, I know this is extremely difficult, but was wondering if it would be possible to assess the impact of month in which you were born on your professional growth / salary growth. Or even more simply, if marks are correlated with professional success. We Indians tend to believe that good marks == success at work. Would love to know if data validates this belief. If only this kind of data were available… Maybe HR teams in organizations could perform such an assessment. Cheers, D.
Vish 16 Aug 2011 9:59 am: http://news.bbc.co.uk/2/hi/8227268.stm Summer-borns tend to lose out at school because, in England and Wales, 1 September is the cut-off date for school entry, so they can be up to a year younger than their classmates. June baby Frank Lampard Research by the Institute for Fiscal Studies two years ago found only 53% of girls and 47% of boys born in August reached the expected educational level at age seven in state primary schools in England, compared with 80% and 70% of those born in September. Their sporting prowess is affected too. Older children tend to be bigger and more confident, receive more encouragement, and their advantage grows over time. For unto everyone who hath shall be given. Among the 25 most capped England football players, 11 were born between September and November, while only one, Frank Lampard, was born between June and August. In his case, having an England international for a father probably outweighed the disadvantage. So for educational and sporting opportunities, the cruellest and kindest months sit next to each other in the calendar like resentful neighbours.

Moderating marks

Sun, 20 Mar 2011 10:23:07 +0000

Sometimes, school marks are moderated. That is, the actual marks are adjusted to better reflect students' performances. For example, if an exam is very easy compared to another, you may want to scale down the marks on the easy exam to make it comparable.

I was testing out the impact of moderation. In this video, I'll try and walk through the impact, visually, of using a simple scaling formula.

BTW, this set of videos is intended for a very specific audience. You are not expected to understand this.

Rough transcript

First, let me show you how to generate marks randomly. Let's say we want marks with a mean of 50 and a standard deviation of 20. That means that two-thirds of the marks will be between 50 plus/minus 20. I use the NORMINV formula in Excel to generate the numbers. The formula =NORMINV(RAND(), Mean, SD) will generate a random mark that fits this distribution. Let's say we create 225 students' marks in this way.

Now, I'll plot it as a scatterplot. We want the X-axis to range from 0 to 225. We want the Y-axis to range from 0 to 100. We can remove the title, axes and the gridlines. Now, we can shrink the graph and position it in a single column. It's a good idea to change the marker style to something smaller as well. Now, that's a quick visual representation of students' marks in one exam.

Let's say our exam has a mean of 70 and a standard deviation of 10. The students have done fairly well here. If I want to compare the scores in this exam with another exam with a mean of 50 and standard deviation of 20, it's possible to scale that in a very simple way.

We reduce the mean from the marks. We divide by the standard deviation. Then multiply by the new standard deviation. And add back the new mean.

Let me plot this. I'll copy the original plot, position it, and change the data.

Now, you can see that the mean has gone down a bit -- it's down from 70 to 50, and the spread has gone up as well -- from 10 to 20.

Let's try and understand what this means.

If the first column has the marks in a school internal exam, and the second in a public exam, we can scale the internal scores to be in line with the public exam scores for them to be comparable.

The internal exam has a higher average, which means that it was easier, and a lower spread, which means that most of the students answered similarly. When scaling it to the public exam, students who performed well in the interal exam would continue to perform well after scaling. But students with an average performance would have their scores pulled down.

This is because the internal exam is an easy one, and in order to make it comparable, we're stretching their marks to the same range. As a result, the good performers would continue getting a top score. But poor performers who've gotten a better score than they would have in a public exam lose out.

Comments

Vinu 13 Apr 2011 8:17 pm: Nice one… your analysis does assume that the performance of a group of students in an easy exam and the performance of the same group of students in a tough exam will both have similar distributions… I am sure you will recall how the loss of resolution in marks (marks were awarded in increments of no less than 1/2) and the clustering of results at the top end ( remember our 10th & 12th public results in ‘90 & ‘92 respy) led to very skewed distribution? In tough exams, I would expect that the skew would shift downward with a lot more positive Kurtosis… so besides the normalization of the exam scores, it would probably be necessary to perform some additional transformations to really compare 2 exams of different ’toughnesses’. All this, without even considering the ‘performance factor’ from the student introducing its own variations… After writing all this, I do realize that my points are perhaps academically motivated - but if all we want to do is to help teachers pick out the students who are more likely to need ’extra tuition assistance’ to make a better showing in a tough public exam, then the basic normalization is sufficient. Of course, most good teachers would say not need statistics to tell them that - they know their students well enough to understand who needs help and who is on autopilot to ace the exams… (ring any bells ? ;) )

Statistically improbable phrases 2

Wed, 14 Feb 2007 12:00:00 +0000

My earlier list of statistically improbable phrases in Calvin and Hobbes is technically just a list of “Statistically Improbable Words”. I re-did the same analysis using phrases. Here are the top 20 statistically improbable phrases (2 - 4 words only):

baby sitter chocolate frosted sugar bombs comic books doing homework fearless spaceman spiff() good night hamster huey ice cream miss wormwood new year peanut butter really think slimy girls spaceman spiff stuffed tiger stupendous man sugar bombs susie derkins watch tv water balloon

That is, these are the 2-4 word phrases whose frequency in Calvin and Hobbes is substantially (at least 5 times) higher than in the other books I have.

While doing this, the single biggest problem that stumped me was: what is a word?

Is “it’s” one word or two words?
Is “six-year-old” one word or three words?
How do I distinguish between abbreviations (g.r.o.s.s.) and full-stops without a space ( … homework.what’s a …)?
Does a comma always split words? (It doesn’t in numbers, like “3,500”)

The other problem is, phrases with more words are more improbable. Right now, if a phrase occurs 5 times more frequently in Calvin and Hobbes than my other books, I include it. But three-letter words rarely occur that often, and four-letter words even less so. Maybe I should have a lower cutoff for longer phrases.

Anyway, this analysis is a crude first approximation. Clearly Amazon’s gotten much further with their system.

Comments

satish 20 Feb 2007 3:56 am: Hey Stud, Satish here, your junior from IIMB. Trying to get in touch with you. Do mail me at satishkgv@hcl.in and let us get in touch.
Oblio 14 Feb 2007 12:00 pm: Fantastic job man! You have unlimited patience!
Reinhard Ebner 14 Feb 2007 12:00 pm: Hey, only just now came across your page, but of the hundreds, if not thousands of C&H sites and tools, this is the most useful I’’ve seen! R
juergwachter 14 Feb 2007 12:00 pm: hello
nice stuff.I intend splitting a text into single words. can you please give me a hint how to do this? I guess there are simple programs doing this.
Many thanks
joe 11 Aug 2009 2:01 am: Nice. Do you have a page where I can try out v2 (phrases)?
The Calvin and Hobbes search Takedown | s-anand.net 21 May 2010 11:53 am (pingback): […] was able to do a lot of cool stuff with this, like statistically improbable phrases and many amusing […]

Not all distributions are normal

Wed, 19 Jul 2006 12:00:00 +0000

14 years ago, I was introduced to the process of normalising grades. Professors “fit” students’ marks into a normal distribution and assign grades based on that. (I still don’t know how they do it).

Since then, I’ve encountered normalising a lot. My performance at work is normalised. I normalise my song ratings and movie ratings. I’ve normalised all kinds of things at work: lead-time of delivery of fans, movements in savings account balances, calls to a call centre, demand for a resource… you name it.

(What I mean by normalising is, I find the mean and standard deviation, and assume that it’s a normal distribution with that mean and standard deviation. For things under my control, like movie ratings, I revise the ratings to fit a normal distribution.)

In fact, I normalise everything I encounter by default.

A few years ago, I started feeling uncomfortable about this. I’ve now figured out why normalising is bad – at least when done blindly like I do.

First, let’s explore why normalising is good. Normalising eliminates biases. If the Prof in Section A grades higher than the Prof in Section B, normalising takes care of it. If a Prof is extremist (more A’s as well as F’s), normalising takes care of it. If a Prof is skewed (lots below average, few extremely high above average), normalising takes care of it.

Eliminating biases makes sense if Section A is fundamentally like Section B. It’s not better, nor more extremist, nor more skewed. If the sections are large enough and picked randomly, this assumption is correct. If Section A represents the smarter half, or people born in the second half of the year, or people from the Western states, or any other non-random selection, this need not be correct.

An aside: You may wonder why people born in the second half of the year is non-random. If school admissions start in September, and admissions start when you’re 3 years old, kids born in September will be nearly 4 years old when they join. Kids born in August will be between just over 3 years. That one-year difference, to a three-year old, is HUGE. For example, you will find a birth date bias in football, with most premiership players being born in the months of September - November.

Normalising goes a step further than eliminating bias, however. Normalising forces a normal distribution. This would be right if the underlying data is normally distributed. But if not, we may be making a mistake by force-fitting.

The Central Limit Theorem says that if you add up random variables, you get a normal distribution. Provided it’s a large sample, variables are independent, and each has a finite standard deviation.

This means that many things you get by adding random variables are normally distributed. For example:

Number of heads when you toss a coin (add up each coin toss)
Average age of an army platoon (add up each soldier’s age)
Terminus-to-terminus time for a bus (add up the time between each stop)
Price movement of an stock exchange index (add up each stock’s price movement)

But a lot of real-life data is NOT normally distributed. The usual reasons are:

It’s not the sum of random variables
It doesn’t satisfy the central limit theorem (independence, large sample, finite standard deviations)

Here are some non-normal distributions that are NOT the sum of random variables:

Soldier’s age within an army platoon. What random variables could you add up? You’ll probably find a lot of people at age 18, because that’s the minimum age. A little fewer at age 19 – last year’s recruits. Far less at age 20 – 2 years minimum service accomplished. Certainly not a normal distribution.
Price movement of a single stock. What random variables could you add up? You’ll find that there are far larger price movements than a normal distribution predicts.

Here are some non-normal distributions that don’t satisfy the central limit theorem. (These are, in fact, things I said were normally distributed earlier. You see? It’s easy to think things are normal, but in reality they’re not.)

The terminus-to-terminus time for a bus. The number of bus stops is quite small. More importantly, the time between stops isn’t independent. If there’s a traffic jam, an entire section of the route will take more time. If there’s a delay between point 2 to 3, it’s likely that there’ll be a delay between points 1-2 and 3-4 as well.
The price movement of a stock exchange index. The price movement of stocks follows a power-law distribution, which does not have finite standard deviations. Also, the price movements are not independent.
See more non-normal distributions.

Summary: Don’t assume that anything you see is a normal distribution. It usually isn’t.

I’ll shortly talk about what happens when you assume something’s a normal distribution, when it really is not.

Comments

Wil 18 Nov 2010 9:14 pm: Great article - you brought it to the point!

Gapminder

Wed, 31 May 2006 12:00:00 +0000

Gap Minder – an excellent dynamic interface to world economic indicators.

Benford Law

Wed, 07 Dec 2005 12:00:00 +0000

Benford’s Law: how to spot fake data.

Google India Zeitgeist

Mon, 21 Feb 2005 12:00:00 +0000

Google India Zeitgeist.

Pizza purchases and tipping

Wed, 31 Dec 2003 12:00:00 +0000

Interesting statistics on pizza purchases and tips.

Coincidences are common

Sun, 14 Oct 2001 12:00:00 +0000

Coincidences are common.

I win page hits in October

Wed, 01 Nov 2000 12:00:00 +0000

I win the bet. ~sanand strikes 14340 to 2931 versus ~mkalidas. Must admit that some people disagree with the majority, though.