probability on S Anand

Bayes’ Theorem

Mon, 11 Oct 2010 14:53:01 +0000

I’ve tried understanding Bayes’ Theorem several times. I’ve always managed to get confused. Specifically, I’ve always wondered why it’s better than simply using the average estimate from the past. So here’s a little attempt to jog my memory the next time I forget.

Q: A coin shows 5 heads when tossed 10 times. What’s the probability of a heads?
A: It’s not 0.5. That’s the most likely estimate. The probability distribution is actually:

That’s because you don’t really know the probability with which the coin will throw a heads. It could be any number p. So lets say we have a probability distribution for it, f(p).

Initially, you don’t know what this probability distribution is. So assume they’re all the same – a flat function: f(p) = 1

Now, given this, let’s say a heads falls on the next toss. What’s the revised probability distribution? It’s:

f(p) ← f(p) * probability(heads | x) / probability(heads) = 1 * (x^1 * (1-x)^0) / 1 = x

Let’s say the next is again a heads. Now it’s

f(p) ← f(p) * probability(heads | x) / probability(heads) = x * (x^1 * (1-x)^0) / 1 = x^2

Now if it’s a tails, it becomes:

f(p) ← f(p) * prob(tails | x) / prob(tails) = x^2 * (x^0 * (1-x)^1) / 1 = x^2 * (1-x)

… and so on. (This happens to be a called a Beta distribution.)

Now, instead of this being the probability of heads, it could be the probability of a person having blood pressure, or a document being spam. As you get more data, the probability distribution of the probability keeps getting revised.

Comments

Ram 18 Jan 2012 1:59 pm: Hi Anand, f(p) ← f(p) * probability(heads | x) / probability(heads) = 1 * (x^1 * (1-x)^0) / 1 = x What I understand from the above notation is f(p) stands for probability distribution function (which is used recursively), x stands for the probability of getting a head (which is unknown) and 1-x stands for the complement of x. Can you please explain why the distribution function is defined as you have done? i.e. , f(p) ← f(p) * probability(heads | x) / probability(heads)

A sense of proportion

Fri, 14 May 2010 08:44:30 +0000

A quote from David Heinemeier Hansson:

So the problem is, a lot of business managers and especially business owners, they have no sense of probability. They can’t fathom that concept. So They treat the probability of 1 to 10 trillion as the same as a 1 to a 100. And like, “We’ve got to deal with this 1 to a trillion probability, because, what if it happens?”

No! Doesn’t matter! I mean, don’t care.

So as soon as that sense of probability spreads, that people can treat that reasonably, I think all this nonsense just goes away.

This lack of proportion, sadly, is at the heart of my every day problems. (Just watch the video!)

Comments

elbin 14 May 2010 5:49 pm: Money matters!

Tamil spelling corrector

Tue, 01 May 2007 12:00:00 +0000

The Internet has a lot of tamil song lyrics in English. Finding them is not easy, though. Two problems. The lyrics are fragmented: there’s no one site to search them. And Google doesn’t help. It doesn’t know that alaipaayudhe, alaipaayuthe and alaipayuthey are the same word.

This is similar to the problem I faced with tamil audio. The solution, as before, is to make an index, and provide a search interface that is tolerant of English spellings of Tamil words. But I want to go a step further. Is it possible to display these lyrics in Tamil?

My Tamil Transliterator does a lousy job of this. Though it’s tolerant of mistakes, it’s ignorant of spelling and grammer. So,

kanda nal muthalai kathal peruguthadi

… becomes…

kanda nal muthalai kathal peruguthadi

… when in fact we want…

kanda naaL muthalaay kaathal peruguthadi

(If you’re viewing this on an RSS reader, check my post to see what I mean.)

I need an automated Tamil spelling corrector. Reading Peter Norvig’s “How to Write a Spelling Corrector” and actually having understood it, I gave spelling correction in Tamil a shot.

Norvig’s approach, in simple terms, is this:

Get a dictionary
Tweak the word you want to check (add a letter, delete one, swap 2 letters, etc.)
Pick all tweaks that get you to a valid word on the dictionary
Choose the most likely correction (most common correct word, adjusted for the probability of that mistake happening)

Making a dictionary is easy. I just need lots of Tamil literature, and then I pick out words from it. For now, I’m just using the texts in Project Madurai.

Tweaking the word to check is easy. Norvig’s article has a working code example.

Picking valid tweaks is easy. Just check against the dictionary.

The tough part is choosing the likely correction. For each valid word, I need the probability of having made this particular error.

Let’s take an example. I’ve spelt kathal. A list of valid tweaks to this word include: kal, kol, kadal, kanal, and kaadhal. For each of these, I need to figure out how often the valid tweaks occur, and the probability that I typed kathal when I really meant one of these tweaks. This is what such a calculation would look like:

Tweak	Frequency	Probability of typing kathal	Product
kal	1	0.04	0.04
kol	4	0.02	0.08
kadal	10	0.1	1.0
kanal	1	0.01	0.01
kaadhal	6	0.25	1.50

Once we have this, we can see that kaadhal is the right one – it has the maximum value (1.50) in the last column, where we multiply the frequency and the probability.

(You probably realise how small my dictionary is, looking at the frequencies. Well, that’s how big Project Madurai is. But increasing the size of a dictionary is a trivial problem.)

Anyway, getting the frequency is easy. How do I get the probabilities, though? That’s what I’m working on right now.

Comments

SDW 1 May 2007 9:01 am: Hi there, with regards to ‘Thendral Ennai Mutham Itathu’ as described on http://www.s-anand.net/blog/classical-ilayaraja/, the female voice was not SPS but B.S. Sasirekha. fyi please.
ரவிசங்கர் 4 May 2007 5:33 pm: நல்ல முயற்சி, அணுகுமுறை.
Ravi 11 May 2007 2:36 pm: Good article. I am very interested in text analysis, phrase extraction too (and statistical machine translation, if you will :) Is not probability just the number of occurences of each of these divided by total number of occurence? Maybe I am missing something. Did you also consider using Vikatan or Kumudam article data in addition to Project Madurai to get contemporary word counts and probabilities? It is one of my projects to create a word frequency list for Tamil. Now that I see another active soul, I might start the project :) Python or ruby I am wondering…
kooliip 1 May 2007 12:00 pm: meaning for saroja sama nikalo
GANESH 1 May 2007 12:00 pm: I LOVE YOU SUJA CHELLAM
hajaroz@yahoo.com 1 May 2007 12:00 pm: please give me that spelling checking software….it is very good and it is new one
v.arul 14 Aug 2008 6:39 am: Give me the correct tamil spelling for unnippaga
shamu 10 Sep 2008 12:17 am: தமிழ்லில் எழுதுவதற்கு முய்ற்சி
ந்ன்றி
nita 29 Dec 2009 3:39 pm: hi i would like to know if there are any resources online which would help me to check if my spellings in tamil typing are right? i am not very confortable with tamil so i can use the help of spell check, i would be most greatful! kindly help, thanks:)
prabhu 13 Aug 2011 8:43 am: Hi anand, Do you have any tamil spell checker tool online. If yes, please provide the link asap. Thanks in advance.
thamizh selvan 19 Mar 2011 4:09 pm: i want to know the exact spelling of thamizh selvan in tamil . whether a connecting letter will come between thamizh and selvan or not .. thank you
chandroo 19 Aug 2012 6:02 am: I need a Tamil spell checking software.Please tell me.
saranraj 1 Jan 2013 9:00 am: Please let me know the correct spelling for the tamil word “Nadathanum”.
Muthu 29 Apr 2015 5:37 am: You may do easy text processing of Tamil content using open-tamil library in Python. $ pip install open-tamil To convert your data into Tamil letters (not Unicode code-points) you can type in Python,

import tamil letters = tamil.utf8.get_letters( data ) Thanks, -Muthu

Not all distributions are normal

Wed, 19 Jul 2006 12:00:00 +0000

14 years ago, I was introduced to the process of normalising grades. Professors “fit” students’ marks into a normal distribution and assign grades based on that. (I still don’t know how they do it).

Since then, I’ve encountered normalising a lot. My performance at work is normalised. I normalise my song ratings and movie ratings. I’ve normalised all kinds of things at work: lead-time of delivery of fans, movements in savings account balances, calls to a call centre, demand for a resource… you name it.

(What I mean by normalising is, I find the mean and standard deviation, and assume that it’s a normal distribution with that mean and standard deviation. For things under my control, like movie ratings, I revise the ratings to fit a normal distribution.)

In fact, I normalise everything I encounter by default.

A few years ago, I started feeling uncomfortable about this. I’ve now figured out why normalising is bad – at least when done blindly like I do.

First, let’s explore why normalising is good. Normalising eliminates biases. If the Prof in Section A grades higher than the Prof in Section B, normalising takes care of it. If a Prof is extremist (more A’s as well as F’s), normalising takes care of it. If a Prof is skewed (lots below average, few extremely high above average), normalising takes care of it.

Eliminating biases makes sense if Section A is fundamentally like Section B. It’s not better, nor more extremist, nor more skewed. If the sections are large enough and picked randomly, this assumption is correct. If Section A represents the smarter half, or people born in the second half of the year, or people from the Western states, or any other non-random selection, this need not be correct.

An aside: You may wonder why people born in the second half of the year is non-random. If school admissions start in September, and admissions start when you’re 3 years old, kids born in September will be nearly 4 years old when they join. Kids born in August will be between just over 3 years. That one-year difference, to a three-year old, is HUGE. For example, you will find a birth date bias in football, with most premiership players being born in the months of September - November.

Normalising goes a step further than eliminating bias, however. Normalising forces a normal distribution. This would be right if the underlying data is normally distributed. But if not, we may be making a mistake by force-fitting.

The Central Limit Theorem says that if you add up random variables, you get a normal distribution. Provided it’s a large sample, variables are independent, and each has a finite standard deviation.

This means that many things you get by adding random variables are normally distributed. For example:

Number of heads when you toss a coin (add up each coin toss)
Average age of an army platoon (add up each soldier’s age)
Terminus-to-terminus time for a bus (add up the time between each stop)
Price movement of an stock exchange index (add up each stock’s price movement)

But a lot of real-life data is NOT normally distributed. The usual reasons are:

It’s not the sum of random variables
It doesn’t satisfy the central limit theorem (independence, large sample, finite standard deviations)

Here are some non-normal distributions that are NOT the sum of random variables:

Soldier’s age within an army platoon. What random variables could you add up? You’ll probably find a lot of people at age 18, because that’s the minimum age. A little fewer at age 19 – last year’s recruits. Far less at age 20 – 2 years minimum service accomplished. Certainly not a normal distribution.
Price movement of a single stock. What random variables could you add up? You’ll find that there are far larger price movements than a normal distribution predicts.

Here are some non-normal distributions that don’t satisfy the central limit theorem. (These are, in fact, things I said were normally distributed earlier. You see? It’s easy to think things are normal, but in reality they’re not.)

The terminus-to-terminus time for a bus. The number of bus stops is quite small. More importantly, the time between stops isn’t independent. If there’s a traffic jam, an entire section of the route will take more time. If there’s a delay between point 2 to 3, it’s likely that there’ll be a delay between points 1-2 and 3-4 as well.
The price movement of a stock exchange index. The price movement of stocks follows a power-law distribution, which does not have finite standard deviations. Also, the price movements are not independent.
See more non-normal distributions.

Summary: Don’t assume that anything you see is a normal distribution. It usually isn’t.

I’ll shortly talk about what happens when you assume something’s a normal distribution, when it really is not.

Comments

Wil 18 Nov 2010 9:14 pm: Great article - you brought it to the point!

Knowing when to stop

Wed, 21 Jun 2006 12:00:00 +0000

Mathematics, marriage and finding somewhere to eat has a simple solution to all these problems. Whether you’re hiring someone, or picking a partner, or finding a house – or any problem that requires you to pick the best among N choices – here’s the rule.

Scan the first 37% of choices. Then pick the first one that’s better than anything you’ve seen so far.

Card trick

Wed, 14 Apr 2004 12:00:00 +0000

Interesting card trick using the Kruskal Count.

More on expectations higher than expected

Thu, 03 Oct 2002 12:00:00 +0000

After reading my post on the ET article mentioning “expected to see a higher than expected rise”, a certain CA gold-medallist friend of mine wrote back this obscure note that I refuse to understand:

… if you take it literally it is not possible. To put it more technically, something called a law of iterated expectation comes to play. Today’s expectation of tomorrow’s expectation about what will happen day after is just today’s expectation of what will happen day after.

I think what they mean by “…” is that the revised expectation is higher than the earlier expectation.

She is in the habit of being right, so I had better accede.

Coincidences are common

Sun, 14 Oct 2001 12:00:00 +0000

Coincidences are common.