transliteration on S Anand

Tamil spelling corrector

Tue, 01 May 2007 12:00:00 +0000

The Internet has a lot of tamil song lyrics in English. Finding them is not easy, though. Two problems. The lyrics are fragmented: there’s no one site to search them. And Google doesn’t help. It doesn’t know that alaipaayudhe, alaipaayuthe and alaipayuthey are the same word.

This is similar to the problem I faced with tamil audio. The solution, as before, is to make an index, and provide a search interface that is tolerant of English spellings of Tamil words. But I want to go a step further. Is it possible to display these lyrics in Tamil?

My Tamil Transliterator does a lousy job of this. Though it’s tolerant of mistakes, it’s ignorant of spelling and grammer. So,

kanda nal muthalai kathal peruguthadi

… becomes…

kanda nal muthalai kathal peruguthadi

… when in fact we want…

kanda naaL muthalaay kaathal peruguthadi

(If you’re viewing this on an RSS reader, check my post to see what I mean.)

I need an automated Tamil spelling corrector. Reading Peter Norvig’s “How to Write a Spelling Corrector” and actually having understood it, I gave spelling correction in Tamil a shot.

Norvig’s approach, in simple terms, is this:

Get a dictionary
Tweak the word you want to check (add a letter, delete one, swap 2 letters, etc.)
Pick all tweaks that get you to a valid word on the dictionary
Choose the most likely correction (most common correct word, adjusted for the probability of that mistake happening)

Making a dictionary is easy. I just need lots of Tamil literature, and then I pick out words from it. For now, I’m just using the texts in Project Madurai.

Tweaking the word to check is easy. Norvig’s article has a working code example.

Picking valid tweaks is easy. Just check against the dictionary.

The tough part is choosing the likely correction. For each valid word, I need the probability of having made this particular error.

Let’s take an example. I’ve spelt kathal. A list of valid tweaks to this word include: kal, kol, kadal, kanal, and kaadhal. For each of these, I need to figure out how often the valid tweaks occur, and the probability that I typed kathal when I really meant one of these tweaks. This is what such a calculation would look like:

Tweak	Frequency	Probability of typing kathal	Product
kal	1	0.04	0.04
kol	4	0.02	0.08
kadal	10	0.1	1.0
kanal	1	0.01	0.01
kaadhal	6	0.25	1.50

Once we have this, we can see that kaadhal is the right one – it has the maximum value (1.50) in the last column, where we multiply the frequency and the probability.

(You probably realise how small my dictionary is, looking at the frequencies. Well, that’s how big Project Madurai is. But increasing the size of a dictionary is a trivial problem.)

Anyway, getting the frequency is easy. How do I get the probabilities, though? That’s what I’m working on right now.

Comments

SDW 1 May 2007 9:01 am: Hi there, with regards to ‘Thendral Ennai Mutham Itathu’ as described on http://www.s-anand.net/blog/classical-ilayaraja/, the female voice was not SPS but B.S. Sasirekha. fyi please.
ரவிசங்கர் 4 May 2007 5:33 pm: நல்ல முயற்சி, அணுகுமுறை.
Ravi 11 May 2007 2:36 pm: Good article. I am very interested in text analysis, phrase extraction too (and statistical machine translation, if you will :) Is not probability just the number of occurences of each of these divided by total number of occurence? Maybe I am missing something. Did you also consider using Vikatan or Kumudam article data in addition to Project Madurai to get contemporary word counts and probabilities? It is one of my projects to create a word frequency list for Tamil. Now that I see another active soul, I might start the project :) Python or ruby I am wondering…
kooliip 1 May 2007 12:00 pm: meaning for saroja sama nikalo
GANESH 1 May 2007 12:00 pm: I LOVE YOU SUJA CHELLAM
hajaroz@yahoo.com 1 May 2007 12:00 pm: please give me that spelling checking software….it is very good and it is new one
v.arul 14 Aug 2008 6:39 am: Give me the correct tamil spelling for unnippaga
shamu 10 Sep 2008 12:17 am: தமிழ்லில் எழுதுவதற்கு முய்ற்சி
ந்ன்றி
nita 29 Dec 2009 3:39 pm: hi i would like to know if there are any resources online which would help me to check if my spellings in tamil typing are right? i am not very confortable with tamil so i can use the help of spell check, i would be most greatful! kindly help, thanks:)
prabhu 13 Aug 2011 8:43 am: Hi anand, Do you have any tamil spell checker tool online. If yes, please provide the link asap. Thanks in advance.
thamizh selvan 19 Mar 2011 4:09 pm: i want to know the exact spelling of thamizh selvan in tamil . whether a connecting letter will come between thamizh and selvan or not .. thank you
chandroo 19 Aug 2012 6:02 am: I need a Tamil spell checking software.Please tell me.
saranraj 1 Jan 2013 9:00 am: Please let me know the correct spelling for the tamil word “Nadathanum”.
Muthu 29 Apr 2015 5:37 am: You may do easy text processing of Tamil content using open-tamil library in Python. $ pip install open-tamil To convert your data into Tamil letters (not Unicode code-points) you can type in Python,

import tamil letters = tamil.utf8.get_letters( data ) Thanks, -Muthu

Google search in Tamil

Fri, 06 Oct 2006 12:00:00 +0000

When I wrote my Tamil song lyrics quizzes, I had two problems:

I can't write in Tamil (not on paper, nor on a computer)
I can't spell right in Tamil (ந vs ன, ர vs ற)

I overcame the first using a Tamil transliterator. I write in English, and you see it in Tamil.

The problem of ந vs ன was simple. ந occurs as the first letter of a word, and just before த. Nowhere else. (Is this always true?)

But ர vs ற can't be solved except through experience, and I'm short of that. So, rather than bother my family with every quiz, I used the wisdom of crowds. I googled both spellings of the word. The correct spelling has more Google hits than the incorrect one.

I did this so often, I made a Google gadget out of it.

Just type the word in English, click 'Search', and my gadget will search in tamil. It's amazing how much stuff there is in Tamil on the Web, from song lyrics to texts (thirukkuraL, for example).

You can add this gadget to:

your desktop (in the Search Gadgets box, type "http://www.s-anand.net/a/tamilsearchgadget.xml")
your website or blog (click here for the code)
Google Reader.

Here's the transliteration table:

Tamil	English
அ	a
ஆ	A or aa
இ	i
ஈ	I or ee
உ	u
ஊ	U or oo
எ	e
ஏ	E
ஐ	ai
ஒ	o
ஓ	O
ஔ	au
க	k or g
ங	n
ச	ch or s
ஜ	j
ஞ	n
ட	t or d
ண	N
த	th or dh
ந	n
ப	p or b
ம	m
ய	y
ர	r
ல	l
வ	v
ழ	zh
ள	L
ற	R
ஷ	sh
ஸ	S
ஹ	h

Comments

Madhu 7 Oct 2006 6:27 am: You should be at google:) and google is now considering buying youtube - remember when i posted a comment once, you said why will they buy:)
S Anand 7 Oct 2006 11:01 am: Yeah, I remember the comment. Good call! Guess YOU’re the one who should be at Google :-)
oskar lewis 7 Oct 2006 12:57 pm: I’ ve put some links on my blog that can help you translating… http://www.oskarlewis.com/weblog/archives/1317
Vasant 7 Oct 2006 5:39 pm: Hi, guess something’s wrong with your RSS feeds.. they don’t seem to reflect your latest posts
S Anand 7 Oct 2006 6:20 pm: Vasant, I subscribe to my own RSS feeds. They seem fine. Are you subscribing to http://feeds.feedburner.com/sanand? Which RSS reader are you using? And what’s the latest post you see?
Vasant 8 Oct 2006 4:43 am: I was subscribing to http://www.geocities.com/root_node/sanand.xml. Works fine with the other URL. Will change - Thanks
Madhu 9 Oct 2006 11:40 am: Not able to download the gadget though. not available in google gadgets as well. would you have a link? Hope google scans ur page and reports that youtube was my idea and look at my email and automatically send me an offer;)
S Anand 9 Oct 2006 12:02 pm: That’s funny, the links in the post work for me… This link should add it to your Google home page. Don’t think you can “download” it, though. You can at best add it to Google Desktop.
Madhu 12 Oct 2006 8:18 am: Ok. Am able to download it as a gadget though. It says add to webpage.
Madhu 17 Oct 2006 11:01 am: This is an interesting website - www.blinkx.com, unlike youtube and google video you can search for a variety of video sources here. Might be bought out by one of the GYM companies soon!
Kesavan.M 30 Oct 2006 10:05 am: Thanks.., No more words
Sudar 4 Jan 2007 10:15 am: It is fine that you did this gadget. I liked it very much. Any such project it pipeline. Can you give this transliteration table in table form?
Sanjai Gandhi 14 Feb 2007 1:53 pm: its a genius thing…amazing…
Sathuragiri vEL 26 Feb 2007 4:30 am: You can have a look at www.tise.velirs.com. We use a simialar approach but has made it more simpler.
saravanan 18 Mar 2007 5:10 am: please help me how to conducts seminar about liver enzymes in liver cancer help from introduction to conclution
ரவிசங்கர் 4 May 2007 5:25 pm: நல்ல Gadget. ஆனா, பலம்-பழம், கள - கல் போன்ற வேறுபாடுகளை அறிய இந்த கூகுள் அணுகுமுறை உதவாது :) தமிழ் அகரமுதலிகள் தான் உதவும். பார்க்க - http://blog.ravidreams.net/?p=152
thuvan 30 Oct 2008 12:29 pm: we can’t convert from Tamil word to English word
tamil priyan 21 May 2009 7:09 pm: Check out http://kandupidi.com/editor . It supports both google tranliteration as well as english phonetic typing as well which many people prefer. There is a gadget for this as well.
John Francis 25 Oct 2011 11:34 am: finding very hard to browse in tamil and i wish to have DEMO SCREEN TO download the steps to use them immeaditely

Making a Tamil transliterator

Mon, 28 Aug 2006 12:00:00 +0000

I've built a simple Tamil transliterator. You can type in words in English and it will spell them out in Tamil. You can copy-paste the Tamil above into Microsoft Word, etc.

You may need to turn on tamil scripts to see the Tamil fonts above. If you have Windows 98, it may not work well. If you've visited this page recently, you will need to refresh this page as well (press F5).

Browse through my Javascript to see how it works. Feel free to reuse.

I've also made a Google Gadget that searches Google in Tamil using this tool.

Here's what to type:

Tamil	English
அ	a
ஆ	A or aa
இ	i
ஈ	I or ee
உ	u
ஊ	U or oo
எ	e
ஏ	E
ஐ	ai
ஒ	o
ஓ	O
ஔ	au
க	k or g
ங	n
ச	ch or s
ஜ	j
ஞ	n
ட	t or d
ண	N
த	th or dh
ந	n
ப	p or b
ம	m
ய	y
ர	r
ல	l
வ	v
ழ	zh
ள	L
ற	R
ஷ	sh
ஸ	S
ஹ	h

I also have a gadget that lets you search in Tamil.

Comments

Arun 31 Aug 2006 8:29 am: Very cool. Why the UI change?
S Anand 31 Aug 2006 10:55 am: Mostly because I wanted to shift focus from “linking” to “writing”.
Sathya 31 Aug 2006 11:06 am: wow … the commenting is back ! great ! In my spare time I would test this and if possible enhance the transtamil. Currently “Aiyya” does not get transliterated and so on.
S Anand 31 Aug 2006 4:19 pm: Try ‘aiyya’ with the small ‘a’. That works. I am allowing only lowercase except for long vowels.
Arun 1 Sep 2006 6:23 am: Ah, ok. Just a suggestion. Maybe you could use up more of the screen width? Lot of screen real-estate is not used currently (at least, on Firefox), so there’s a lot of scrolling (down) to do.
S Anand 1 Sep 2006 7:13 am: Arun, I thought a lot about this. While this makes it tougher on people who visit my site a lot, most newcomers find a linear flow easier. Since 90% of the traffic to my site is newcomers (most regular readers use RSS), I thought it best to design it this way. Just curious – what do you scroll down for? The links?
Arun 1 Sep 2006 11:55 am: I scrowl down for the links, yes. Actually, I don’t check all of them out when I first see your updates. Usually, return 2-3 times a day, when I am more in the mood to read. I sometimes tend to return to your old posts 4-5 days later. Also, I was a bit confused about the separation between the links and your writings. I didn’t realize that till you explained in your previous comment. Actually, I see your updates on google reader, but for whatever reason, I check your page too. Don’t know why I do that! :-)
sathish 21 Sep 2006 3:59 am: very nice anand. Thank you.
Vidya 6 Oct 2006 5:42 am: how can i get sri?
Vidya 6 Oct 2006 5:45 am: got it.. Dont bother to reply
RMurali 20 Nov 2006 7:06 pm: Dear Anand, I got to know your site while searching London on youtube videos. you are doing a good job. keep it up !
Victor 17 Jan 2007 10:21 pm: try aeiou as a word. remains as aeiou. Seems to be a method to the madness since these are all vowels
venkatachalam 18 Jan 2007 4:46 pm: thsi is very nice.can you please tell me which free software help me to do this transliteration.pl mail to bvchalam1946@gmail.com
Siva Prasadh 12 Apr 2007 3:28 pm: Its very useful i like it very much
senthil 28 Aug 2006 12:00 pm: its good buddy.. very easy…
sridhar 28 Aug 2006 12:00 pm: Superappu…..!
sriram 28 Aug 2006 12:00 pm: how do you write the character ‘akk’? (Aaytha ezhuthu), as in ‘fa’ inn faathima?
meenakshi 28 Aug 2006 12:00 pm: great tool!! checked it and worked very well, only issue i had was i could not figure out how to get the nya working, as in kavinyar.
Kavitha 28 Aug 2006 12:00 pm: Ur site is simply excellent. Very very useful for Tamilians….Tamil Patru Ullavanga across the world.
ராஜ் தொழில்நுட்பம்.com 28 Aug 2006 12:00 pm: வணக்கம் ஐயா,
தமிழ் எண்கள் க, உ ஆகியவை செயல்படுத்தச்செய்தால் மிக நன்று. இருந்தாலும் மிக அருமை.
இங்ஙனம், ராஜ்
thozhilnutpam.com; geocities.com/tamildictionary
RAAJARAM K 28 Aug 2006 12:00 pm: மிக அருமை. இண்டெர்னெட் ல் இல்லாத போது உபயோகிப்பது எவ்வாறு என்று தெரிவித்தால் நன்றாக இருக்கும்.

Sanskrit transliterator

Mon, 28 Aug 2006 12:00:00 +0000

I’ve built a simple Sanskrit transliterator. You can type in words in English and it will spell them out in Sanskrit. You can copy-paste the Tamil above into Microsoft Word, etc.

Browse through my Javascript to see how it works. Feel free to reuse.

Comments

ND 28 Aug 2006 12:00 pm: Real good and absolutely quick!
Saurabh 28 Aug 2006 12:00 pm: Good work, wonder how many use Sanskrit these days though! One more thing, specific to Mac users, is that Indic support is still not mature for Firefox. Safari, the inbuilt browser does a decent job though. There was an extra “S” that was appearing after each letter in Safari.
Madhu 28 Aug 2006 12:00 pm: Awesome, again what ur doing at Infy consulting, u shud be developing products at google:)
KK 28 Aug 2006 12:00 pm: It is really awesome
vikram 28 Aug 2006 12:00 pm: i cannot see any translation in firefox 1.5.0.12 version
Animesh 28 Aug 2006 12:00 pm: Good tool, but the transliteration scheme should be replaced by more standard one.
Ashwin (you know me!!) 28 Aug 2006 12:00 pm: nice one buddy!!although how would you deal with ambiguities like “om” ?
Anand 28 Aug 2006 12:00 pm: Hmm, I wonder…can you please check if the small ‘‘i’’ maatra works fine? I gave the word ‘‘janani’’ and the ‘‘i’’ gave me the maatra AFTER the second ‘’na’’, while it should be BEFORE. The big ‘‘ii’’/‘‘I’’ works fine. Also words like ‘‘chithram’’ etc - basically wherever there is a small ‘‘i’’!
Rama Krishna 28 Aug 2006 12:00 pm: Really Good. Keep up the good work
mohana 28 Aug 2006 12:00 pm: try this
swami neelkanth 28 Aug 2006 12:00 pm: thanks