perl on S Anand

Perl, 1994-2011

Sat, 28 Sep 2024 16:51:14 +0000

In 1994, I learnt Perl. It was fantastic. I used it to:

1995: Build CCChat - the unofficial IITM email system and software repository
1999: Build my entire blog from scratch
2000: Author my 2nd year thesis on the Behavioural Aspects of Financial Analysts by analyzing 600MB of IBES data
2002: Analyze where to place the central processing hubs for a bank
2004: Analyze the interest durations of public sector banks
2005: Creating music quizzes
2006: Create my own music search engine (which earned me about $100 a month in Google Ad revenue for a while)
2006: Automated resume filtering
2007: Create custom search engines
2008: Build application launchers

In 2006, I was convinced I should stick to Perl over Python.

In 2008, Google launched AppEngine and it provided free hosting (which was a big deal!) but had only 2 runtimes: Java and Python. The choice was clear. I’d rather learn Python than code in Java.

By 2011, I stopped installing Perl on my laptop.

Though most people know me mainly as a Python developer, I’ve programmed in Perl for about as long as I have in Python. I have fond memories of it. But I can’t read any of my code, nor write in it anymore.

When I watched The Perl Conference (now called The Perl and Raku Conference – Perl 6 is called Raku), I was surprised to hear how much the language had declined.

There were fewer than 100 attendees - and for 2025, they’ve decided to go smaller and book a tiny hotel, so as to break-even even if only 20 people show up.

Few languages have had as much of an impact on my life and thinking. My knowledge of modern programming comes from The Camel Book, functional programming from Higher Order Perl, Windows programming from Learning Perl on Win32 Systems, and so on. Even my philosophy of coding was shaped by Larry Wall’s the three great virtues of a programmer.

This is my homage to the language that shaped me. Bless you, Perl!

The Social Network

Thu, 24 Feb 2011 19:00:33 +0000

4:00pm. Just started watching The Social Network. I’m fairly sure I won’t like the film, mostly because I’ll be jealous of Mark.

About 5 minutes into the movie. I find myself rewinding to catch the dialogues. They’re very fast. Very, very fast.

10 minutes. I like the code. I stopped on a screen to start checking if it’s real code. It’s in Perl. Stopped myself before I started dry-running the code.

20 minutes. OK, some of this is a bit beyond me. But good movie.

1:20. Dropped a note to Ram asking (almost forcing) him to watch the film. A fair bit to learn from a stakeholding perspective.

1:45. Dammit. I see a bit of the anti-social me in him.

Gripping. Overall, bloody good movie. Quite surprised. And no, definitely not feeling jealous.

To Python from Perl

Tue, 20 Jan 2009 12:00:00 +0000

I’ve recently switched to Python, after having programmed in Perl for many years. I’m sacrificing all my knowledge of the libraries and language quirks of Perl. The reason I moved despite that is for a somewhat trivial reason, actually. It’s because Python doesn’t require a closing brace.

Consider this Javascript (or very nearly C or Java) code:

var s=0;
for (var i=0; i<10; i++) {
    for (var j=0; j<10; j++) {
        s = s + i * j
    }
}

That’s 6 lines, with two lines just containing the closing brace. Or consider Perl.

my $s = 0
foreach my $i (1 .. 10) {
    foreach my $j (1 .. 10) {
        $s = $s + $i * $j
    }
}

Again, 6 lines with 2 for the braces. The $ before the variables also drops readability just a little bit. Here’s Python:

s = 0
for i in xrange(1, 10):
    for j in xrange(1, 10):
        s = s + i * j

On the margin, I like writing shorter programs, and it annoys me to no end to have about 20 lines in a 100-line program devoted to standalone closing braces.

What I find is that once you’ve really know one language, the rest are pretty straightforward. OK, that’s not true. Let me qualify. Knowing one language well out of C, C++, Java, Javascript, PHP, Perl, Python and Ruby means that you can program in any other pretty quickly – perhaps with a day’s reading and a reference manual handy. It does NOT mean that you can pick up and start coding with Lisp, Haskell, Erlang or OCaml.

Occasionally, availability constrains which programming language I use. If I’m writing a web app, and my hosting provider does not offer Ruby or Python, that rules them out. If I don’t have a C or Java compiler on my machine, that rules them out. But quite often, these can be overcome. Installing a compiler is trivial and switching hosting providers is not too big a deal either.

Most often, it’s the libraries that determine the language I pick for a task. Perl’s regular expression library is why I’ve been using it for many years. Ruby’s HPricot and Python’s BeautifulSoup make them ideal for scraping, much more than any regular expression setup I could use with Perl. Python Image Library is great with graphics, though for animated GIFs, I need to go to the GIF89 library in Java. And I can’t do these easily with other languages. Though each of these languages boast of vast libraries (and it’s true), there are still enough things that you want done on a regular basis for which some libraries are a lot easier to use than others.

So these days, I just find the library that suits my purpose, and pick the language based on that. Working with Flickr API or Facebook API? Go with their default PHP APIs. Working on AppEngine? Python. These days, I pick Python by default, unless I need something quick and dirty, or if it’s heavy text processing. (Python’s regular expression syntax isn’t as elegant as Perl’s or Javascript’s, mainly because it isn’t built into the language.)

To get a better hang of Python (and out of sheer bloody-mindedness), I’m working through the problems in Project Euler in Python. For those who don’t know about Project Euler,

Project Euler is a series of challenging mathematical/computer programming problems that will require more than just mathematical insights to solve. Although mathematics will help you arrive at elegant and efficient methods, the use of a computer and programming skills will be required to solve most problems.

Each problem has been designed according to a “one-minute rule”, which means that although it may take several hours to design a successful algorithm with more difficult problems, an efficient implementation will allow a solution to be obtained on a modestly powered computer in less than one minute.

It’s a great way of learning a new programming language, and my knowledge of Python is pretty rudimentary. At the very least, going through this exercise has taught me the basics of generators.

I’ve solved around 40 problems so far. Here are my solutions to Project Euler. I’m also measuring the time it takes to run each solution. My target is no more than 10 seconds per problem, rather than the one-minute, for a very practical reason: the solutions page itself is generated by a Python script that runs each script, and I can’t stand waiting for more than that to refresh the page each time I solve a problem.

Comments

Nicola 17 Dec 2014 10:15 am: Hi guy, I think you should add a comment form on your Euler page… this is not the right page for this comment. Anyway it seems your page is one of the favourites for the students that have project Euler probems as class assignments. So I’d like to contribute… This is a better solution for the problem #18 using an easy formulation od Dijkstra algorithm. tr = ( (75, ), (95, 64, ), (17, 47, 82, ), (18, 35, 87, 10, ), (20, 4, 82, 47, 65, ), (19, 1, 23, 75, 3, 34, ), (88, 2, 77, 73, 7, 63, 67, ), (99, 65, 4, 28, 6, 16, 70, 92, ), (41, 41, 26, 56, 83, 40, 80, 70, 33, ), (41, 48, 72, 33, 47, 32, 37, 16, 94, 29, ), (53, 71, 44, 65, 25, 43, 91, 52, 97, 51, 14, ), (70, 11, 33, 28, 77, 73, 17, 78, 39, 68, 17, 57, ), (91, 71, 52, 38, 17, 14, 91, 43, 58, 50, 27, 29, 48, ), (63, 66, 4, 68, 89, 53, 67, 30, 73, 16, 69, 87, 40, 31, ), ( 4, 62, 98, 27, 23, 9, 70, 98, 73, 93, 38, 53, 60, 4, 23, ), ) sums=[0] for i in xrange(len(tr)): ns=[0]*len(tr[i]) for c,v in enumerate(tr[i]): if c>0: s0=sums[c-1]+v if s0>ns[c]: ns[c]=s0 else: pass try: s1=sums[c]+v if s1>ns[c]: ns[c]=s1 except: pass sums=ns max(sums) For each row the program stores the greater accumulator of the numbers leading to that row, so it is not an O(2^n) problem but a O(n^2). I have also a solution for problem #31: coins = (1, 2, 5, 10, 20, 50, 100, 200) count=0 def coinFlip(level, residual, pattern): global count, tests p = residual / coins[level]
print level, residual, pattern, innResid
while True: tests += 1 innResid = residual-p*coins[level] if level>0: coinFlip(level-1, innResid, pattern + (p,)) if p==0: break p=p-1 else: if innResid==0: count += 1
print count, pattern + (p,)
break count=0 tests=0 coinFlip(7, 200, ()) It works by “changing” the coins: we start with one 2P and change it for two 1P, then we change one of the two 1P with two 50p… and so on. I’d be very glad if you like them and decide to publish my code on your site. Greetings, Nicola.

Caching pages on Apache

Thu, 14 Aug 2008 12:00:00 +0000

I don’t use any blogging software for my site. I just hand-wired it some years ago. When doing this, one of the biggest problems was caching.

Consider each blog entry page. Each page has the same template, but different content. Both the template and content could be changed. So ideally, blog pages should be served dynamically. That is, every time someone requests the page, I should look up the content, look up the template, and put them together.

I did that, and within a few days outgrew my hosting service’s CPU usage limit. Running such a program for every page hit is too heavy on the CPU.

One way around this is to create the pages beforehand and serve it as regular HTML. But every time the template changes, you need to re-generate every single page. I had over 2,500 pages. That would kill the CPU usage if I changed the template often.

At that point, I did a piece of analysis. Do I really need to regenerate all 2000 blog entries? Wouldn’t the 80-20 rule apply? The Apache log confirmed that 20% of the URLs were accounting for 76% of the hits. So I’d be wasting my time regenerating all the pages every time I changed the template.

So based on this, I decided to dynamically cache the pages. When a page is requested for the first time, I create the page and save it in a cache. The next time, I’d just serve it from the cache. If the template changes, I just need to delete the cache. This way, I only generate pages that are requested, and they’re only generated once.

OK, so that’s the background. Now let me get to how I did it.

I wrote a Perl script, blog.pl, that would generate a page in the html folder whenever it is called. Next, I changed Apache’s .htaccess to run this program only if the page did not exist in the html folder.

# Redirect to cache first
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^/]*)\.html$       html/$1.html

# If not found, run program to create page
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^html/([^/]*)\.html$  blog.pl?t=$1

The first block redirects Apache to the cache. The second block checks if the file exists in the cache. If it doesn’t, the Apache redirects to the program. The program creates the page in the cache and displays it. Thereafter, Apache will just serve the file from the cache.

This Apache trick can be used in another way. I keep files organised in different folders to simplify my work. But to visitors of this site, that organisation is irrelevant. So I effectively merge these folders into one. For example, I have a folder called a in which I keep my static content. I also have this piece of code:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^/]+)$   a/$1

If any file is not found in the main folder, just check in the a/ folder. So I can access the file /a/hindholam.midi as </hindholam.midi> as well.

This can be extended to a series of folders: either as a cascade of caches, or to merge many folders into one.

Comments

Kannan Ekanath 17 Aug 2008 7:00 am: [http://random-thoughts-and-rambling.blogspot.com/]

Hi, I wanted to know a comparison/contrast how setting up blog etc in your own hosted space compares with things like http://sites.google.com. I have always postponed this buying of a domain in favour of things like sites.google.com. Obviously I understand that the advantages would be you can control the LAF, the content, the organisation etc. But have you got a comparison doc or something on what precisely one would get if he hosts things in his own domain instead of a google/yahoo powered wiki style page?
S Anand 17 Aug 2008 1:31 pm: @Kannan: Don’t have anything offhand, Kannan. But buying a domain would make sense anyway – you could point your domain to sites.google.com transparently. If you wanted to build your own blogging system for your site, though, I’d advise against it. I did it because I started in 1999 and there weren’t many blogging systems then. Today, it just doesn’t make sense.
Nirupam 18 Aug 2008 7:43 am: Hi Want to create a site for myself. Should I go for own hosted space or should I start using sites like Geosites etc. I want to deploy web-based applications in my site. Can you please advise?
S Anand 19 Aug 2008 11:24 am: @Nirupam: If you want to deploy your own applications, you should probably create your own site. Get a free hosting service to start with. I’ve used 110mb.com, freestarthost.com and awardspace.com, and found them to be OK.

Launching applications

Sat, 02 Aug 2008 12:00:00 +0000

Opening programs from the Start - All Programs menu is painful. For many years, I relied on the quick launch bar.

But it’s space constrained. There are only so many applications you can place there. I want space enough for frequently used documents as well. Recently, I decided that I need all the space on the screen. So my task bar is on auto hide, and that makes the quick launch bar a little tougher to use as well. And finally, I can’t use the quick launch bar with the keyboard. That’s important.

So I switched to the pinned menus on the Start Menu.

This works better with the keyboard. I access Word, I just type the Ctrl-Esc, W. Excel: Ctrl-Esc, E. But I run short of letters soon. I have trouble between Powerpoint and processing, for instance. And I can’t store documents.

I tried Enso Launcher and Launchy, both of which are great products, but I just can’t stand the thought of them hogging up all the memory that they do. Launchy in particular.

Given that I almost always have one or two command prompts open, I write my own little tool to do the job now. It’s a command line launcher I’ve written in Perl. I call it “o”. At the first run, it indexes my hard disk. (Well, not all of it. I’ve picked what I need.) Now, if I want to read Harry Potter and the Deathly Hallows, I just type:

> o harry potter hallows

If I wanted to pick a Harry Potter book, I could:

> o harry potter
    0: D:/Entertainment/Books/Hugo Awards/2001 - J K Rowling - Harry Potter and the Goblet Of Fire.rar
    1: D:/Entertainment/Books/J K Rowling.1.Harry Potter and The Sorcerer's Stone.pdf
    2: D:/Entertainment/Books/J K Rowling.2.Harry Potter and The Chamber of Secrets.pdf
    3: D:/Entertainment/Books/J K Rowling.3.Harry Potter and The Prisoner of Azkaban.pdf
    4: D:/Entertainment/Books/J K Rowling.4.Harry Potter and The Goblet of Fire.doc
    5: D:/Entertainment/Books/J K Rowling.5.Harry Potter and the Order of the Phoenix.pdf
    6: D:/Entertainment/Books/J K Rowling.6.Harry Potter and the Half-Blood Prince.pdf
    7: D:/Entertainment/Books/J K Rowling.7.Harry Potter and the Deathly Hallows.pdf
    8: D:/Entertainment/Books/J K Rowling.The Harry Potter Encyclopedia.doc
    9: D:/My Pictures/2005-06 London/2005-07-16 06 Waterstones Oxford Street Harry Potter release.JPG
    ... more
> (0-9, q, any word): prince
D:/Entertainment/Books/J K Rowling.6.Harry Potter and the Half-Blood Prince.pdf

The program lists the files matching the words I typed, and lets me filter within that.

I just wrote this yesterday, and already, I’ve used it dozens of times. Here’s the source.

PS: While I was at it, I downloaded a Flickr uploader for Perl. So I can now upload images with the command line. This easily saves me at least 5 minutes per article.

Comments

Karthik A 2 Aug 2008 12:00 pm: Oc here
I got equally bugged long back and wrote a python program called fo (folder opener ;-)

Here it is - crude but worked pretty well \
@setlocal enableextensions & python -x %~f0 %* & goto :EOF \
import os
import ConfigParser
import sys

CONFIGFILENAME = “config.ini”
CONFIGFILENAME1 = “config1.ini”
cleanInp = ‘’
scuth = {}
\
Load all the config file details in a hash \
cfgFile = ConfigParser.ConfigParser()
cfgFile.read(CONFIGFILENAME)
cfgFileFP = open(CONFIGFILENAME,‘r+’)

scuts = cfgFile.items(“shortcuts”)

\
load into a dictionary \
for (x,y) in scuts:
scuth[x]=y

print “““Folder and file rapid opener (FO)
————————————
Ctrl-Alt-C - to start
@shortcut,path - to add a shortcut
x - exit
————————————”””

while cleanInp != ‘x’:
inp = raw_input(“Your wish Master!:”)
cleanInp = inp.strip()
if cleanInp.find(’@’) == -1:
try:
os.startfile(scuth[cleanInp])
except:
pass
else:
tmp = cleanInp.split("@")[1]
tmp1 = tmp.split(",")
cfgFile.set(“shortcuts”,tmp1[0], tmp1[1])
cfgFile.write(cfgFileFP)
q
Sathish 2 Aug 2008 12:00 pm: Anand,

Did you try slickrun.. It is extremely slick..
S Anand 2 Aug 2008 12:00 pm: @OC: Hey, didn’t realise you were into python. Neat!

@Sathish: How much memory does Slickrun use up? Given some of the apps I’m running on my 1GB laptop, I’m having to count each byte!
oracle 2 Aug 2008 12:00 pm: The method I use:

enable google desktop by ctrl-ctrl

in the search bar first few alphabets of the app..invariably it shows me the app i need.
Saurabh 2 Aug 2008 12:00 pm: I used to use Launchy before switching to Mac an year book and have been using QuickSilver since then. Dont use the Apple Spotlight, neither the Google Desktop thing..QS does all the work - launching apps, searching for files and quickly accessing them, a nice article abt a great QS trick (for Mac users) here : http://www.43folders.com/2005/06/13/quicksilver-the-comma-trick

BTW Processing.org seems to be interesting, plan to check it out soon!

Cheers!
S Anand 11 Aug 2008 5:35 am: @Sathish: 7MB is not too bad. Launchy was taking up over 25MB in my case.

Reducing the server load

Wed, 06 Jun 2007 12:00:00 +0000

I’m been using a shared hosting service with 100 WebSpace over the last 7 years. It’s an ad-free account that offers 100MB of space and 3GB of bandwidth per month. Things were fine until two months ago, which was when my song search engines started attracting an audience. I had anticipated that I might run out of bandwidth, so I used a different server (that has 5GB of bandwidth per month quota) for loading the songs. But what I didn’t anticipate whas that my server load would run over the allotted CPU limit.

You’d think this is unusual, given how cheap computing power is, and that I’d run out of bandwidth quicker. But no. My allotted limit was 1.3% of CPU usage (whatever that meant), and about 2 months ago, I hit 1.5% a few times. I upgraded my account to one which had a 2.5% limit immediately, but the question was: why did this happen?

This blog uses a lot of Perl scripts. I store all articles on a MySQL database. Every time a link is requested, I dynamically generate the HTML by pulling up the article from the MySQL database and formatting the text based on a template.

I also use MySQL to store users’ comments. Every time I display each page, I also pull out the comments related to that page.

I can’t store the files directly as HTML because I keep changing the template. Every time I change the template, I have to regenerate all the files. If I do that on my laptop and upload it, I consume a lot of bandwidth. If I do that on the server, I consume a lot of server CPU time.

Anyway, since I’d upgraded my account, I thought things would be fine. Two weeks ago, I hit the 2.5% limit as well. No choice. Had to do something.

If you read the O’Reilly Radar Database War Stories, you’ll gather that databases are great for queries, joins and the like, while flat files are better to process large volume data as a batch. Since page requests come one by one, and I don’t need to do much batch processing, I’d gone in for a MySQL design. But there’s a fair bit of overhead to each databasse query, and that’s the key problem. Perl takes a while to load (and I suspect my server is not using mod_perl). The DBI module takes a while to load. Connecting to MySQL takes a while. (The query itself, though, is quite fast.)

So I moved to flat files instead. Instead of looking up from a database, I just look up a test file using grep. (I don’t use Perl’s regular expression matching because regular expression matching in UNIX is faster than in Perl.) I have a 1.6MB text file that contains all my blog entries.

But looking up a 1.6MB text file takes a while. So I split the file based on the first letter of the title. So this post (Reducing the server load) would go under a file x.r.txt (for ‘R’) while my last post (Calvin and Hobbes animated) would go under a file x.c.txt (for ‘C’). This speeds up the grep by a factor of 5-10.

On average, using MySQL query used to take 0.9 seconds per query. Now, using grep, it’s down to about 0.5 seconds per query. Flat files reduced the CPU load by about half. (And as a bonus, my site has no SQL code. I never did like SQL that much.)

So that’s why you haven’t seen any posts from me the last couple of weeks. Partly because I didn’t have anything to say. Partly because I was forced to revamp this site.

Comments

Sathya 6 Jun 2007 12:00 pm: Wow. thats quite interesting. Perhaps you could have also tried caching. keep a copy of ‘‘hot’’ (Most Recently used -MRU) stuff in memory, so you dont have to relaod it again. Given that most of the hits would be for the latest entries ( or atleast 80% of the hits are for 20% of ur posts) then caching might be an option. However, I dont know how much of RAM quota you have ;-)
somnath 6 Jun 2007 12:00 pm: Do you work at your company in business consulting or technology. Having known you for the past eight years (jeez a long time) I have no doubt you manage not just both but spend quality time with your family as well. Maybe it would be a good idea to blog about your typical day when you juggle between work (consulting for a big client), passion (revaming this site or taking pictures or doing a project like building media pc etc. etc.) and family (playing with a baby daughter is wonderful). BTW got your mail but I check internet mail very infrequently. Cheers!
Saurabh 6 Jun 2007 12:00 pm: This one is slightly off the topic..I also started using freehostia a few months back, however I find that 2 out of 7 days in a week, my website is not available for more than 96% of the time..(I monitor the uptime using something called as InternetSeer)..Would you have any tips here, is freehostia not a good option for free services or should I switch to some other service provider?
Rocco Blegen 4 Dec 2011 4:14 am: Howdy! This post couldn’t be written any better! Reading through this post reminds me of my old room mate! He always kept talking about this. I will forward this post to him. Pretty sure he will have a good read. Thank you for sharing!

Splitting a sentence into words

Thu, 26 Apr 2007 12:00:00 +0000

I often need to extract words out of sentences. It’s one of the things I used to build the Statistically Improbable Phrases for Calvin and Hobbes. But splitting a sentence into words isn’t as easy as you think.

Think about it. What is a word?

Something that has spaces around it? OK, let’s start with the simplest way to get words: split by spaces. Consider this piece:

"I'd look at McDonald's," he said.
"They sell over 3,000,000 burgers a day -- at $1.50 each."
High-fat foods were the rage. For e.g., margins in fries
were over 50%... and (except for R&M & Dyana [sic]) everyone
was at ~30% net margin; growing at 25% too!

Splitting this by spaces (consider new lines, tabs, etc as spaces too.), we get the following:

"I'd
look
at
McDonald's,"
...

Now, some of these like “I’d” are words. But “McDonald’s” isn’t. I mean, there’s a full-stop and a double-quotes at the end. Clearly we need to remove the punctuation as well. But, if we do that, I'd becomes Id. So we need to be careful about which punctuation to remove. Let’s take a closer look.

The following punctuation marks are clear word separators: spaces, the exclamation mark, the question mark, semicolon, brackets of any kind, and double-quotes (not single quotes). No word has these in the middle. If we use these as separators, our list of words is better, but we still have some words with punctuation:

McDonald's,
e.g.,
High-fat
R&M
...

The issue is, these punctuation marks are ambiguous word separators: comma, hyphen, single-quote, ampersand, period and slash. These usually separate words, but there are exceptions:

Comma: Not inside numbers: 3,000,000.
Hyphen: Not for hyphenated words: High-fat.
Single-quote: Not for possessives: McDonald’s. Not for joint words: I’d.
Ampersand: Not for abbreviations: R&M
Period: Not for abbreviations: O.K. Not for URLs: www.s-anand.net
Slash: Not for fractions: 3/4. Not for URLs: google.com/search

Colon is ambiguous too. In normal English usage, it would be a separator. But URLs like http://www.s-anand.net/ use these characters, and it doesn’t make sense to separate them.

So here are my current rules for splitting a sentence into words. (It’s a Perl regular expression. Don’t worry. Cooper’s Law: If you do not understand a particular word in a piece of technical writing, ignore it. The piece will make perfect sense without it.)

# Split by clear word separators
/       [\s \! \? \;\(\)\[\]\{\}\<\> " ]
 
# ... by COMMA, unless it has numbers on both sides: 3,000,000
|       (?<=\D) ,
|       , (?=\D)
 
# ... by FULL-STOP, SINGLE-QUOTE, HYPHEN, AMPERSAND, unless it has a letter on both sides
|       (?<=\W) [\.\-\&]
|       [\.\-\&] (?=\W)
 
# ... by QUOTE, unless it follows a letter (e.g. McDonald's, Holmes')
|       (?<=\W) [']
 
# ... by SLASH, if it has spaces on at least one side. (URLs shouldn't be split)
|       \s \/
|       \/ \s
 
# ... by COLON, unless it's a URL or a time (11:30am for e.g.)
|       \:(?!\/\/|\d)
/x;

This doesn’t even scratch the surface of the issue, though. Here are some issues:

Lots of files split words into two at the end of a line. How do we handle that?
How do we handle incorrect punctuation? For instance, if someone types “done.Yet,” without leaving a space after the full-stop, I’ll think it’s an abbreviation.
What about other separators? Like the ± symbol or the £ symbol for instance.
What about other languages?!

And you thought it was easy!

Comments

Kasi 27 Apr 2007 10:18 pm: Hi Anand, Nice site..It was good talking to you! When u find time, pls do visit my humble less visited blog at www.raconteurkasi.blogspot.com !
Ramya 3 May 2007 7:52 am: Hi Anand, Can u guide me through the interview process of INSEAD MBA . Can you share your wisdom with me to crack the interview, and one more question .. do u sleep for less than 5 hrs a day ?
Andrew 26 Apr 2007 12:00 pm: Thanks, this is very helpful.

Hindi songs online

Mon, 19 Feb 2007 12:00:00 +0000

Click here to search for Hindi songs. This is an article on how I wrote the search engine.

I find it a nuisance to have to go to Raaga, search for a song, not find it, then go to MusicIndiaOnline, not find it, then go to Musicplug.in, and so on until Google.

So I got the list of songs from some of these sites, put it together in one place, and implemented a find-as-you-type.

Just go to s-anand.net/hindi and type a song or movie name.

Update: How I created this is a very long story, spanning over two years. And here it is.

Over the last 5 years, my MP3 collection had grown quite large. Most of these were from MP3 CDs I had bought, songs I’d downloaded, or songs I’d recorded. In any case, the file names were a mess. In early 2005, I decided to organise the collection and put them all into one big directory per language, and name the files in the format “Movie.Song.mp3”.

People think I’m crazy to use a single large directory. But I prefer one directory with 5,000 files to 1000 directories with 5 files for a simple reason. Searching in one directory is easier than in multiple directories. You can just sort everything by name, date modified, size, whatever. On the command prompt, you can type “DIR dil*.txt” to see all movies starting with “Dil”.

I chose the “Movie.Song.mp3” format because the movie name and the song name were really the only two things I knew about every song I had. I didn’t always know the music director, singers or year of the movie. And I placed “Movie” before “Song” because I often browse songs within a single movie, and it’s useful to sort by name in Windows Explorer and see all the songs in a movie. I’ve never had a need to sort by song name. If I wanted to find a song that started with, say, “pehla”, I’d just type “DIR *pehla*” on the Command Prompt. (As you might have guessed, I have a Command Prompt window always open.)

So, having very rationally organised my music collection, I was happy.

Soon the problem shifted to song discovery. I’d heard the familiar songs in my collection many times. Of the unfamiliar songs, but I didn’t know which to pick. I knew I liked some music directors more than others, and had a hunch I liked older songs. (My subsequent analysis of song preferences confirmed this.) But I didn’t know the year or music director for any of my songs.

Since Raaga had a decent collection of songs, along with the year and music director, I decided to download this information and tag my files with this information. There were two problems. Firstly, the data in Raaga needs to be parsed. I need to hunt through each file to find the year and music director. The second was worse: my song names were spelt differently from Raaga’s.

Step 1: download the HTML and parse it. Perl is pretty much the only programming language I know. I used Perl’s LWP library to download the movie index of Raaga. Each movie in the index always has the URL http://www.raaga.com/channels/hindi/movie/something.html. So I extracted these patterns and downloaded all these URLs as well. (Others have a recognisable pattern as well: http://www.musicindiaonline.com/music/hindi_bollywood/s/movie.some_number/, http://www.musicplug.in/songs.php?movieid=some_number, http://ww.smashits.com/music/hindi_film/songs/some_number, etc.)

You probably realise that I downloaded a fair bit of the entire Raaga.com site’s HTML. Actually, it’s not that big. The 800-odd files in the Hindi collection didn’t take more than 40MB of space, and a few hours to download. Here’s what the code looks like:

# Get the list of movies HTML file
my $index = get("http://www.raaga.com/channels/hindi/movies.asp");

# Extract the list of movies from that into a hash (movie, url pairs)
my %movie = ($index =~ m|<a href="/channels/hindi/movie/([^"]*).html" class="[^"]*">([^>]*)</a>|gsi);
 
# Loop through each movie
for my $file (keys %movie) {
  # Get the song HTML file
  my $html = get("http://www.raaga.com/channels/hindi/movie/$file.html");
 
  # Year is typically like this: Movie_Name (1983)
  my $year = ($html =~ m|<b>$movie{$file}\s+\((\d*)\)</b>|) ? $1 : "";
 
  # Music director always has the URL /channels/hindi/music/something
  my $md = ($html =~ m|<a href="http://www.raaga.com/channels/hindi/music/[^"]*" class="black">(.*?)</a>|) ? $1 : "";
  for (split /\n/, $html) {
 
    # Now get individual songs and rating. They always have an onClick="setList(something)"
    if (m|onClick="setList1\((\d*)\)[^>]*>(.*?)<\/a>.*?"http://images.raaga.com/.*?.gif" alt="RATING: ([\d\.]*)"|) {
    $song = $2;
    $rating = $3;
    print join("\t", $movie, $song, $year, $md, $rating), "\n";
  }
}

Incidentally, I’m showing you a simplifed version. I actually wrote the script in a way that I could resume where I left off. The ability to resume was probably the single most useful time-saver in the entire process.

Step 2: match the song names with those on my list. This is tricky. I hate doing it manually. So I developed a set of rules that could compare two spellings of a movie and decide if they were the same or not (see my earlier post on matching misspelt movie names). For Hindi songs and movies, here are my rules (in JavaScript):

w = w.toUpperCase(); // Convert to upper case
w = w.replace(/\s+/, " "); // All whitespaces = 1 space
w = w + " "; // Ensure there's a space after every word
w = w.replace(/W/g, "V"); // V=W
w = w.replace(/Z/g, "J"); // Z=J
w = w.replace(/PH/g, "F"); // PH=F
w = w.replace(/([KGCJTDPBS])H/g, "$1"); // Ignore H after most consonants
w = w.replace(/(.)\1/g, "$1"); // Ignore repeated letters
w = w.replace(/Y/g, ""); // Ignore Y
w = w.replace(/([AEIOU])N /g, "$1"); // Ignore trailing N after vowel (hein, mein)
w = w.replace(/[AEIOU]/g, ""); // Ignore vowels
w = w.replace(/ /g, ""); // Ignore spaces

These are the rules, incidentally, that I use in my Hindi quizzes. Even if you type in a different spelling, the rules above will match the correct answer.

I extended these programs over 2006 to cover MusicIndiaOnline, Musicplug.in and Smashits. (I’ve hit a point of diminishing returns, I think, so I’m not too keen on expanding this list.)

Now, with a database of song information, I needed a good media player to view this in. I’ve used several media players over time: WinAmp, Windows Media Player, RealPlayer, iTunes, and MediaMonkey. I’m a big WinAmp fan, but I’ve been forced to Media Monkey now. WinAmp has a 10 second delay before playing any song on my new laptop. MediaMonkey’s not bad, though. The lack of advanced filters is countered by the heavy programmability (I can use Javascript to update MP3 ID3 tags on MediaMonkey). Plus, I get all the WinAmp plugins. As for the other media players, I think they’re junk.

There are five things I want in a perfect media player:

Find as I type. I shouldn’t have to type the entire song, or press a “Go” button. I’ll just type. It should show all matches instantly. WinAmp does this, and that’s why I loved it. (Today, most media players can do this.)
Advanced filters. Instead of manually creating playlists, I’d rather create filters, like “Highly rated songs in the 2000s I haven’t heard recently”. (See How I listen to music.)
Enqueable playlists. When I click on a song, I don’t want my current song to be interrupted. Just play it next.
Global hotkeys. I want to pause the song when someone says something – without having to go to the application, search for the pause button, etc. WinAmp does this brilliantly with its global hotkeys.
Online and offline integration. I want to be able to search online collections, like Raaga. Unfortunately none of the media players can do this. They have their own collections (radio stations, really), but even these aren’t really searchable.

Since my favourite media players (WinAmp and MediaMonkey) lack only one of these features, I thought I might be able to build them in.

But no such luck. It was almost easier to build my own media player. So I started to build my two weeks ago. My hope was to cover as many of my favourite requirements, beginning with find as you type.

The key to find-as-you-type is speed. You can’t afford many calls back and forth between the browser and the server. Even if people have a fast connection, my server is not fast enough. (A good part of the reason why I use Google applications is speed. Google’s server is blazingly fast, and the design of their applications complements that.) To make find-as-you-type fast, ideally the entire database should be loaded. Then, as you type, I just need to check with the database in memory. But downloading an entire database takes ages! (My full music database is 7MB right now.)

Step 3: compress the database. Rathern than load the full 4MB, I managed to get the page to start after loading 100KB of data. First, I cut down less important fields. Most searches are for a song or movie, rarely for a year or music director. So I took only the movie and song names. That brought the data down to ~2MB.

I don’t need to repeat the movie name across songs. If I have something like this:

Movie1  Song1
Movie1  Song2
Movie1  Song3
Movie2  Song1
Movie2  Song2

I can store this instead as:

Movie1  Song1   Song2   Song3
Movie2  Song1   Song2

I can also eliminate duplicate names. This brings down the space requirements to ~500KB.

The next step was the clever one. I don’t need to load the full database before you start searching! It’s enough to load a reasonable subset, and let you start searching while the rest loads in the background. So my hindi song search engine loads about 100KB of the data from one Javascript file, lets you search, and in the background loads the 400KB Javascript file. As soon as that finishes loading, it displays results from that set as well. (The initial portion is really songs on Raaga. I figured it would represent a decent search set.)

Step 4: find-as-you-type. This is quite easy, actually. I send the onkeyup event to my search function.

<input id="inp" onkeyup="csearch()">

The csearch() function goes through all the songs, checks if there’s a match, and prints all those results.

// Create a case-insensitive regular expression
var re = new RegExp(search, "i");
for (var i in songs) {
  if (re.test(songs[i)) { str += songs[i]; }
}
library.innerHTML = str;

But that, unfortunately, takes ages to finish. If you implemented this as is, you would have to wait 1 - 1.5 seconds between each key stroke. So I made two modifications. Firstly, I restrict the number of results displayed to 100. When you start typing, (for example, you’d go ‘c’… ‘ch’… ‘chu’…) there are several songs early on that match the string, so I don’t have to search through the whole list. This speeds up the search for small strings.

When the search gets bigger, (‘chupke chu’… ‘chupke chupk’…), there aren’t 100 results. So the search has to cover the full list, and that takes 1-1.5 seconds between each keystroke. So I took another clever step. I broke the search into chunks of 5000 songs. That takes a fraction of a second. I search successive chunks of 5000 songs. If I find any results, I add them. Then I wait for a keystroke. If nothing happens, I continue searching the next 5000 after 50 milliseconds, and so on. If you press a key in the meantime, I stop searching for the original word, and start searching for the new word. This makes the application appear a lot faster.

There are ways I could make this even faster. For example, people type more often than delete. A typical sequence would be (‘chupke ch’… ‘chupke chu’… ‘chupke chupk’…) rather than the reverse. Backspace is not pressed very often. So, instead of re-searching the whole list, I could just search the already-searched list in such cases. But right now, the search works fast enough, so I’ll leave it at that.

The next step is advanced filters. I’m working on that right now. Hopefully you’ll see in a while.

Comments

reshma 21 Feb 2007 9:46 am: your search tool is awesome !
sarangan 21 Feb 2007 9:18 pm: anand - can you illuminate the technique you used to consolidate the names of tracks in one list and the dynamic search based on combination of letters ?
gagan 21 Feb 2007 10:42 pm: god level tool man! awesome!! please do elaborate on the methodology followed to make the tool. thanks :)
S Anand 22 Feb 2007 2:22 pm: Sure. Will post the details soon.
Sai 23 Feb 2007 4:27 am: Well, now that you’ve posted this list, you should become the next raaga and allow songs to be heard!
S Anand 23 Feb 2007 7:38 am: You can do that even now – just click on the song :-)
jake 23 Feb 2007 9:17 am: dude your tool is good.
Saurabh 23 Feb 2007 2:21 pm: Brilliant! Sheer genius!
Sai 23 Feb 2007 4:45 pm: Yup i just realized!! Mighty impressive bud!
Sriram 23 Feb 2007 11:45 pm: thamizh sevai..
shane 24 Feb 2007 11:34 am: i love you
Karthik A 25 Feb 2007 2:04 am: This is awesome man. 2 days I have seen this I dont go to Music India or Raaga. This is neat, but how are you going to keep it updated?
Raga 25 Feb 2007 6:24 am: dude, is it possible to download songs like ilayaraja’s how to name it?.. please tell how
S Anand 25 Feb 2007 8:59 am: Hi Karthik, updating is not an issue. I have an automated script that checks for updates and downloads the info. Takes 10 minutes. I’ll run it every two weeks or so.
S Anand 25 Feb 2007 9:00 am: Raga, try TamilTorrents.net – they have the full How To Name It collection.
Ramkumar R 25 Feb 2007 9:08 am: Awesome tool! Really appreciate the effort.. Just a suggestion.. CoolToad has a huge tamil/hindi collection as well.. Any possibility of indexing that as well?
Raga 25 Feb 2007 10:50 am: Thanks anand… i couldnt find the songs there. there is no searching tool there.. please post any other place where i can find them.. thanks once again
S Anand 25 Feb 2007 11:45 am: Ramkumar, cooltoad is next on my list.
Raga 25 Feb 2007 2:04 pm: One more thing anand, there are many very good BGm tracks by ilayaraja.. is it possible to add them ? rhanks
S Anand 25 Feb 2007 2:08 pm: Raga, you’re right – I couldn’t find it on TamilTorrents any more. I have a bunch. Mail me at root dot node at gmail dot com. And where can I find links to the background music collection? Would love to add them
Ravi 25 Feb 2007 2:49 pm: thanks for the useful tool. you rock!
Karthikeyan 25 Feb 2007 6:04 pm: Well done Anand! The tool is amazing!
Karthikeyan 25 Feb 2007 8:11 pm: See my blog! I have published an article about this
Divya 25 Feb 2007 8:55 pm: aaamazing!!!!
Divya 25 Feb 2007 9:00 pm: bookmarked.. added to favourites.. everything :)
Divya 25 Feb 2007 9:01 pm: i love the part where the results start appearing even before u finish typing.. like the google toolbar search
senthil k 25 Feb 2007 9:49 pm: nice one
Meenakshi Sundaram 25 Feb 2007 10:32 pm: Anand, You have a amazing job man. Keep it up.This would help many tamil music lovers and save their time on searching. I am really greatfull to you for the effort and time you have put.
johan paris 25 Feb 2007 10:58 pm: fantastic job
Prasanna 25 Feb 2007 11:02 pm: This is amazing. Well done!
Thiagu 25 Feb 2007 11:59 pm: really good tool Anand. Thanks for the same. I am not able to download any songs. rightclick and “save target as” is saving the html page. Could you provide your suggestion
SP.VR.Subbiah 26 Feb 2007 1:27 am: I appreciate your marvelous work Many songs in tamil are available in www.musicplug.in Site. Please provide your links also to that site
keshav bhat 26 Feb 2007 3:10 am: Fantastic! I’m not certain how you found time for this, but the results are truly fantastic! keep up the great work
Radha 26 Feb 2007 4:03 am: Thanks so much. You are genius.
Jala 26 Feb 2007 4:50 am: Really cool
sundararajan 26 Feb 2007 5:26 am: Anand fantastic buddy…beautiful…..
sundar 26 Feb 2007 5:27 am: Anand simply superb..GR888
naveen 26 Feb 2007 5:37 am: nice tool
usha 26 Feb 2007 5:45 am: a very good job anand, came thro karthiks blog. i have been trying for a long time to download the song thendral vanthu theedum pothu from avatharam. though i couldnt download i was happy to hear the song here. thanks
sathik 26 Feb 2007 6:24 am: coll saite
Rakesh 26 Feb 2007 6:53 am: Its wonderful .. and ultra fast .. Thanks.
Priya 26 Feb 2007 10:10 am: nice site enjoyed a lot thanks
Gops 26 Feb 2007 11:50 am: wonderful stuff from u Anand..really a nice one.. i’ve been searching songs from Thalaivasal Album all these days… wow finally…. thanks a lot…..
Kaps 26 Feb 2007 2:43 pm: Anand, This is a great tool. Thanks for all the effort. Can you also include Tamilbeat and Thenisai into this?
Chakra 26 Feb 2007 3:05 pm: Brilliant stuff Anand. Thank you.
Kaps 26 Feb 2007 3:08 pm: Anand, Understand your time constraints. Just wanted to give some hint about the ways to strengthen this tool. IMHO, Music Director - good to have (not essential). Year - doesn’t matter that much to music fans (low priority). The percentage of people wanting to know the year would definitely be low.
S Anand 26 Feb 2007 3:53 pm: Thanks, Kaps. It’s exactly this sort of prioritisation help that I need. Will try and pull in these additional databases.
Cosmic Voices 26 Feb 2007 3:53 pm: Truly amazing!!!
Rajagopalan 26 Feb 2007 4:27 pm: Hi Anandavi Kannnadasan,s songs are available in Tamilnation website Why don’t u combine this in the Tamil songs script. V.Rajagopalan Chennai
Syam 26 Feb 2007 5:59 pm: Awesome search engine dude….Thanks bunch
thetalkativeman 26 Feb 2007 7:08 pm: Anand, hats off to you. This is fantastic :) I’ve been using these sites for 5 years on a daily basis and your tool means a lot to me :-)
kittu 26 Feb 2007 8:37 pm: nice work anand. key change search is great. GR8 work.
kittu 26 Feb 2007 8:43 pm: you are going to rock the music world maamu :-)
madhu 27 Feb 2007 1:15 am: that is cool. thanks
Kaps 27 Feb 2007 2:55 am: Anand, You are right, the collection in Thenisai and Tamilbeat are not all that impressive. Most of them are recent collections and they would have been covered by Raaga and Music India anyway.
Adiya 27 Feb 2007 5:29 am: Super .romba nalla eruku.. super super
Mugunth 27 Feb 2007 6:46 am: Hi Anand, Wonderful service. I have blogged about this here: http://mugunth.blogspot.com/2007/02/tamil-song-search.html
S Anand 27 Feb 2007 7:03 am: Thanks, everyone! Enjoy.
deekshanya 27 Feb 2007 10:40 am: Your effort needs true appreciation! Good job! Hatsoff for all the hardwork
Gopinath Selvaraj 27 Feb 2007 11:47 am: This is the first ever search engine that gave me immediate results for songs like ‘Aayar paadi maaligayil’ and ‘meenkodi theryil’… Its a google for Tamil and Hindi Music. Thanks a lot for your hard work.
Viren Manek 27 Feb 2007 1:00 pm: I liked this search very much. Its fantastic….
Viren Manek 27 Feb 2007 1:04 pm: Why dont u keep the hit counter on ur site, u will get lots of hit…. :). People will like to bookmark this URL for any song to listen.
Jaganath 27 Feb 2007 4:35 pm: Unbeleivable Performance of the Search Functionality. Cool Site dude!…
Raamcm 27 Feb 2007 4:40 pm: You did great job man…. Thanks
ரவி 27 Feb 2007 7:03 pm: chanceless tool !!!! thank u so much..i was fed up clicking link after links in raaga, musicindia etc..please give more tools like this for the needs of tamils, indians. நன்றி
Kapil 27 Feb 2007 10:12 pm: Anand, AWESOME Tool ! I found songs that I never thought I’d find. However, most of the links were not working. A few took me to either raaga or musicplug but there were more that gave an error message saying the link on sanand.freehostia.com is not working
Prabha 28 Feb 2007 1:39 am: Great piece of work, Anand. I always wanted to have such a tool for Tamil music. In fact, i started to write something similar, more of a user contribution kind of thing, similar to IMDB. Displaying other Meta tags (artist, movie, year, director) would be really useful here. I appreciate your work!
Saurabh 28 Feb 2007 6:32 am: Addition of playlist by Music Directors is awesome!
cr.sathish 28 Feb 2007 5:28 pm: Dear anand, its really kool and kudos to you for making this up.
madhan 28 Feb 2007 6:57 pm: u r rocking…
reshma 28 Feb 2007 8:32 pm: anand, would it be too much to ask for multiple selections against a list of songs?
RPanda 28 Feb 2007 8:42 pm: Hi Anand, it’s a very neat site with great feature. Really appreciate your efforts!!
S Anand 28 Feb 2007 9:19 pm: Reshma, what do you mean by multiple selections?
Maya 28 Feb 2007 9:52 pm: Great job ..
buspass 1 Mar 2007 1:13 am: cool work man. AJAX rocks.
Vinay 1 Mar 2007 2:57 pm: Excellent.
Sriram 1 Mar 2007 7:14 pm: Great Tool!! Awesome Job!!
Supremus 1 Mar 2007 10:06 pm: Awesome job dude!
reshma 1 Mar 2007 10:26 pm: right now i click on a single song, hear it, and come back here to click on another. It would help if I could select more than one song at a time - so that those songs could play one after the other without my intervention.
S Anand 1 Mar 2007 10:55 pm: Reshma, continuous playback is what I want next as well. It’s very difficult, though. Don’t know if I’ll be able to do it. Will try.
maniprakash 2 Mar 2007 2:55 am: Good work man,
aman 2 Mar 2007 3:00 am: pretty cool man
aman 2 Mar 2007 3:01 am: y not create a request play… like a playlist created by everyone… which everyone can hear
BHARANI 2 Mar 2007 7:39 am: This looks brilliant. Good Concept and congrats.
Anonymous 3 Mar 2007 9:57 am: unfin!!!finally u got it man!!!!really super work! mercy…add more top 20 in all language tamil,english hindi…mercy….
nagu 3 Mar 2007 12:43 pm: அருமையான படைப்புங்க….ரொம்ப நன்றிங்க….வெறும் நன்றிங்கறத விட நாலுபேருக்கு இத இன்ட்ரோ… கொடுத்தாத்தான் கரெக்ட்டா இருக்கும் என்ன நாஞ்சொல்றது…. சரிதானுங்களே?
PK 4 Mar 2007 6:16 am: Good work dude…
Tejas 4 Mar 2007 7:29 pm: I salute your creative/techi mind.Thats AJAX right ?
shema 5 Mar 2007 12:08 pm: Few suggestions of movies to add to your playlist. Vennira Aadai (Enna enna vaarthaigalo), Kunguman (Chinnanj chiriya Vanna paravai), Paadhai theriyudhu paar (Thennangeetru oonjalile) I’ve searched for these songs in the net and have failed.
shema 5 Mar 2007 12:10 pm: Excellent work. I see the desi Indian instincts being very strong in you. Did this work take a vigorous shape after Shilpa’s issue?
S Anand 5 Mar 2007 1:16 pm: Hema chithi, the songs are already there. Just type Vennira Aadai ~ Enna Enna Varthaigal, Kumkumam ~ Chinnanchiriya and Pathai Theriyudhu Paar ~ Thennan Keethu Onjalile. Who is Shilpa?
indu 5 Mar 2007 4:47 pm: hi anand, cool site… you should add a hits counter, you could rake some big bucks with ads if this link gets popular..
தமிழ்பித்தன 6 Mar 2007 2:15 am: உண்மையில் உங்கள் முயற்சி என்னை மெய்சிலிர்கக வைக்கிறது உங்கள் தயவால்விரும்பிய பாடலை நின்ற இடத்திலிருந்து கேட்க முடிகிறது கொசுறு:-இன்னும் வேறு தளங்களையும் இணைக்க இயன்றளவு முயலுங்கள்
Anonymous 6 Mar 2007 5:05 am: கலக்கல் சாமீ ஆனந்த் அருமையான உழைப்பு வாழ்த்துகள
रवि 6 Mar 2007 6:43 am: बहुत उम्दा प्रयास और सराहनीय कार्य. हमारी शुभकामनाएँ.
Debashish 6 Mar 2007 9:00 am: Splendid work guys :)
Sagar Jain 6 Mar 2007 9:38 am: Good work, keep it up :)
shema 6 Mar 2007 11:32 am: Shilpa Shetty. The actress who stood up against racial discrimination.
shema 6 Mar 2007 12:15 pm: I could hear thennangeetru……. but unable to listen to chinnannchiriya vanna paravai. Help me with other search words. Also the song chinna chinna mookuthiyaam from the same movie paadhai theriyudhu paar
sri 6 Mar 2007 6:08 pm: you have done agreat job man
srk 7 Mar 2007 9:10 am: Anand, there is a good collection of songs on dhingana.com , specially marathi, can you index those as well ?
Ravi 8 Mar 2007 6:30 pm: hi anand, is there any chance of making this as a toolbar which can be downloaded in to IE or Mozilla?
ram 8 Mar 2007 10:05 pm: hey man..gr8..keep going…it wud be best if u can giv downloads too
Subhash 10 Mar 2007 3:47 pm: Anand Bhai You should be no.1 on “The Google” list. You are the best.
Hamish 10 Mar 2007 4:16 pm: man thanx for ths great site.it helps really.keep it up.kudo
Ashish Ranpara 11 Mar 2007 10:17 am: plz keep going and get more and more old songs you have a really good old hindi songs i really glad for that thanks……
Sriram 11 Mar 2007 4:10 pm: ur tool is just awesome!!!!!
Saravanan 11 Mar 2007 7:04 pm: Dear Friends, I am a big fan of tamil comedy dramas by Shekar and Crazy Mohan and talk show by Leoni. Is there anyway I use this tool to find the availabile mp3s on that?
Sanketh 12 Mar 2007 8:11 am: Man, this is a truly cool tool … Can you do the same thing for Tamil/Telugu songs? Tell me something - does Raaga etc show information about how long the song is? I can think of a really cool tool if raaga does maintain that info.
S Anand 12 Mar 2007 8:44 am: Sanketh, none of the sites show the song length, unfortunately. But if we could get it, what could we do?
Mano 12 Mar 2007 3:11 pm: great work Anand !!
Sanketh 12 Mar 2007 6:56 pm: We could create an online playlist of songs, drag and drop songs from your search to a frame on the right. To play we just load the link in a hidden iframe (or a frame at the bottom). The problem is we wouldn’t know when each song is done to switch to the next song.
Anonymous 12 Mar 2007 8:17 pm: This is amazing Anand .. I wish there was one for Telugu/Tamil songs as well.
S Anand 13 Mar 2007 12:21 am: Sanketh, good point. I haven’t given up hope on that, though. May still be possible. Let’s see…
vijay 13 Mar 2007 6:25 am: Great work anand
Anotheranon 13 Mar 2007 4:29 pm: Anand ! Love ur work! I am looking for “Nee munnale pona naan pinnaale vaaren” REMIX from vaathiyaar. Any idea why not there ?
S Anand 13 Mar 2007 5:33 pm: Search for “Yennadi Muniyamma”, and you’ll find the song.
Mahendra 13 Mar 2007 8:00 pm: Ananda i want to know, can i download the songs that r being played after i click on them & when realone player window opens
S Anand 13 Mar 2007 8:53 pm: Not on this page, Mahendra. Go instead to s-anand.net/hindimp3 or s-anand.net/tamilmp3 to download songs.
rajvir 14 Mar 2007 5:26 am: i want to download the song..jeene ke ishaare…but it ’s not there in your content please help me
S Anand 14 Mar 2007 6:03 am: Rajvir, the song is there in the collection. When you tried it, for some reason, it was using only a fraction of the index. Just wait a few seconds for the full index to load (or reload the page), and you’ll have the song.
mak 14 Mar 2007 8:46 pm: infact a good tool. but cudn’t find movie “begunaah qaidi”. neways thumbs up to u.
sundar 15 Mar 2007 1:39 pm: Excellent work, Anand!!! Thank you very much
sundar 15 Mar 2007 1:52 pm: Anand, not sure if this is already on cards. It would be great if you can add more Carnatic/Tamizh isai songs to the database
Raja.S 16 Mar 2007 5:21 am: Hey Anand, Excellent man.. I got rare songs I was looking for a long time. Thanks a ton. Love Raja
Sanketh 16 Mar 2007 5:26 am: Man, your search algo is pretty decent too. Was just searching for ’elangaathu’ and typed it as elankaa.. and it still showed the right song.
S Anand 16 Mar 2007 9:16 am: Sundar, Carnatic music search is already live at s-anand.net/carnatic and I’m planning to add to the (currently somewhat sparse) collection.
Kalpesh 16 Mar 2007 11:49 pm: I really liked the site & idea. I am not sure that I am being naive, is it possible for me to select multiple songs & play them one by one?
Smitha 17 Mar 2007 2:20 pm: Excellent site very cool.
S Anand 17 Mar 2007 3:46 pm: Kalpesh, playing multiple songs is not possible right now. I’m trying. It’s difficult, though. Don’t know if I’ll be able to do it. Will keep trying.
மதி 17 Mar 2007 7:15 pm: அண்ணன் என்னடா தம்பி என்னடா…. பாடலைக் கேட்டேன் இதனை எப்படி தரவிறக்கம் செய்து கேட்பது எனத் தெரியவில்லை, அருமையான பாடலைத் தந்தமைக்கு மிக்க நன்றி
S Anand 17 Mar 2007 8:30 pm: Mathy, search for “Annan Ennada” at s-anand.net/tamilmp3
Krishnamurthy 18 Mar 2007 5:05 pm: you worked very very hard make such a search tool, its really amazing keep well , keep doing good work
Sundar 21 Mar 2007 3:26 am: Anand, it would be great if we can have some site preferred over others (something like appearing on Top). Because as I observed oosai & Music-Plugin has got good sound quality over others. I also found my favorite carnatic songs in the carnatic page. Thanks again!!
Chirag Mehta 21 Mar 2007 9:16 am: Hey Anand , Really Good Site I liked it What was ur idea behind this Site development is the same which i was facing. Thanx for a ready made solution for me Added Link to ur Site as a post on my site http://www.chiragmehta.info/chirag/2007/03/21/hindi-songs-online-search-engine/
S Anand 21 Mar 2007 11:37 am: Sundar, how about this order: MusicPlug.in > Oosai > Musicindiaonline > Raaga > Smashits?
Shekhar 22 Mar 2007 4:34 pm: Good site. I am searching for koyil pura film song ‘vedam nee’ n surfed the whole net without success. Please help
S Anand 22 Mar 2007 5:15 pm: Shekhar, search for “koyil pura” on s-anand.net/tamilmp3 and you’ll find Vedham nee
Shekhar 22 Mar 2007 8:05 pm: Thanks Anand. I tried for almost an hour but could not able to download the song as it always says the network is busy
S Anand 22 Mar 2007 8:10 pm: Shekhar, I’m having some trouble at cooltoad as well. Usually, I wait a day and it’s fine.
Shekhar 23 Mar 2007 6:28 pm: Tried whole day but without luck. Any other option.
Aqua 23 Mar 2007 6:49 pm: aWesoME TOOL.
sundar 24 Mar 2007 7:39 am: Anand, Thanks for considering my request. That order would be great.
Varun 24 Mar 2007 10:50 pm: Hi! Awesome search…just like Winamp…lot of effort put..good going!!
shilpa 25 Mar 2007 9:12 am: hi! Congrats…. amazing search. Could you add other regional songs to the list??
பாலச்சந்தர் முருகானந்தம் 26 Mar 2007 8:51 am: நல்ல கருவி
Prasun 28 Mar 2007 4:49 am: I came here searching for music. Your tool is awesome. It would be nice if this were extended to playlist creation etc - I think it would probably be not very hard if all songs are from a single site. Oh, and I spent a couple of hours in your writings section!
Varun 29 Mar 2007 11:14 am: Great tool anand!
Seshan 29 Mar 2007 11:59 am: Anand, just happened to visit this page. Awesome. Couple of songs that I found it difficult to get were made so easy with your tool. (Sorgame endralum, solladi bharatha maatha to name a couple). Great work.
Seshan 29 Mar 2007 12:51 pm: Anand - For carnatic songs, can’t the search be made on ‘ragam’. It would be cool to have it. Just a suggestion.
Roopa 29 Mar 2007 4:21 pm: Hi!!…Superb job! Good going..
Anonymous 30 Mar 2007 8:54 am: hai friends.. its not working
Shekhar 30 Mar 2007 9:46 am: After continuous effort I managed to download koyil pura songs. Many thanks Anand for making my day. Keep up the great job
Vishal 31 Mar 2007 1:51 pm: Anand ,I have forgotten all other hindi song websites now.
Venkat 2 Apr 2007 6:29 am: Good keep it up I advocate open source can u think of simple tools for day to day money management and account management with banks
Kannan 5 Apr 2007 9:56 am: First of all good work !!! I keep referring to this whenever I have to get some movie names for songs. I personally have a 17Gig songs collection, I would like to know if you have encountered an API to change properties of a mp3 file like title, album title etc. programmatically (preferably in Java or Groovy or Ruby or Perl) ???
abbas 8 Apr 2007 11:16 am: good job man,thanks alot.
Anu 9 Apr 2007 12:24 am: Hi Anand…this is really a great job…i need a help…i find that tamil and hindi songs are available at your site for download…Can you please help me out with the carnatic songs too….for now it would be great if i have ‘Sachmara ramavani’ atleast….i have been trying to get this song for such a long time…thanks so much for the great work!
Viju 10 Apr 2007 6:48 am: Interesting work. Is there a similar serach tool for carnatic/hindi radio? There is a similar search site for lyrics…at lyricsdir.com.
ramkrishna 11 Apr 2007 4:09 am: hindi song
ramkrishna 11 Apr 2007 4:11 am: Kachche Dhaage’s songs
Kavitha 12 Apr 2007 2:51 am: That was a wonderful tool Anand. I found it very useful… :)
Pradeep 12 Apr 2007 3:29 am: hello bro awesome work da. can u pls find me a song from Prem Qaidi - I live for u.
jagannath 13 Apr 2007 11:03 am: awesome job, Anand could you also upload kannada. Will You
Nainesh 15 Apr 2007 12:25 am: Superb, and fast i like the idea, Keep it up, i found it useful.
sameer jain 15 Apr 2007 4:56 am: awesome anand but can please help me in finding one song from maine pyaar kiya ..its aate jaate i am not able to hear it ..please help .. but one thing is for sure ..u really did a gr8 job ..get this gud work going all the way up ..all the best
Anonymous 16 Apr 2007 6:20 am: awesome job…..
Manikandan 18 Apr 2007 10:59 am: I guess there are some issues. Webpage is loaded with errors. Also, Why cant we have Punnagai Mannan Theme Music?
animesh 19 Apr 2007 8:38 am: Hey Anand….u are genius…great idea…nicely implemented…..all of us (i.e. people like me) are greatful to u for making this tool and letting us use it…..one idea (if u have some free time) that u can try ….let users create a playlist using ur site….
Hemant B. 20 Apr 2007 5:14 pm: Thanks Annand you have created very useful site for music lovers. wish you best luck for your new creation.
jitendrasharma 22 Apr 2007 12:53 pm: jitu
thuva 22 Apr 2007 1:48 pm: wow nice and great i will give you 90% for this website tthat tamil sitet downloading for free i love it and your the best fro my life ok then take care and good luck best of luck and great and great good site and 90% rating
your name 27 Apr 2007 12:05 am: awesome collection!!
SHAMIT 27 Apr 2007 7:48 am: Absolutely splendid job using: Perl, your brain and its ideas … !! ;)
SHAMIT 27 Apr 2007 1:38 pm: Can’t stop admiring! Killer App. Listening to Guide songs BRILLAINT !!
Sabarish 28 Apr 2007 4:10 pm: Its absolutely wonderful…..Really helpful for avid song listeners like me..the web application speaks for itself…i dont need to add any more nice adjectives to it. great work dude..
Deepak 30 Apr 2007 8:46 am: great job anand, really great collection
Siva Chandran P 3 May 2007 11:17 am: Great Job
mahesh raghunathan 4 May 2007 10:10 am: really great da… infact i was searchin for SRI RAGAVENDRA movie scorings for a long time.. got it frm ur site… its really good… keep going..
Anonymous 5 May 2007 3:50 am: great job anand..searching a song is very easy now..hats off to u.
k.Bharathi Kannan 5 May 2007 5:34 am: First of all, THANKS A LOT to you sir. Me too had disappointed in searching for many songs.But I dont know any sites to which i hav to search for songs. This is the first site which i came to know about, regarding cine songs. And this is really great work. Once again THANKS A LOT sir………
your name 6 May 2007 4:16 pm: Hi…will you be making the carnatic songs too downloadable? am waiting for them…..
avinash 8 May 2007 6:44 pm: great work anand.. its really very simple to listen songs over here. can u please include the songs from strings..the pakistani band. thanx for creation of such link…keep goin
shivam 12 May 2007 11:21 pm: thank you so much yaar i am simply feeling so relaxed now. after searching for geeta dutt songs for six months i finally got them from your site. thank you so much and if there is anyway i can send songs to upload, do tell me.
srinath 13 May 2007 6:47 am: hi dude mann u r search tool i awesome and its really so simpel to get us the required song. One suggestion to u buddy….how abt adding a playlist…so tht ppl can add their list and den keep njoyin the music……
srikan 15 May 2007 5:49 pm: gud one. u can actually make it further responsive. do u use prototype?
Bala 17 May 2007 8:38 am: Your site is the single most impressive site i have seen.. Kudos http://quietturbulence.blogspot.com
sai kalyan 17 May 2007 11:27 am: excellent site man really great search tool
mohan 18 May 2007 7:10 pm: Nice job dube :)
shashwat 19 May 2007 12:05 pm: Great job buddy !! Thanks a lot
InTeGeR 20 May 2007 9:22 am: Simply awesome… You helped me get some old Ilayaraja gems. Thanks a lot. It has also climbed up in Google search
Ravi 19 Feb 2007 12:00 pm: Nice Job Anand! Excellent work!!
thiru 19 Feb 2007 12:00 pm: song is good but how can i download plz give me the link thak for the song
Paritosh Arya 19 Feb 2007 12:00 pm: Hi Anand, You really have done a lot of research while developing this search engine….really inspiring…keep it up.
venkat 19 Feb 2007 12:00 pm: Excellent job man.hats off to ur idea.One question is that cant we select multiple songs at a time and play the continously becos i think thats repeated song to play a single song.if we can do that pls tell me how to do that
rama ragbir 19 Feb 2007 12:00 pm: thanks for a great service . i am looking for a song of
Jagadesh Kumar 19 Feb 2007 12:00 pm: Hi Anand This is a fantastic effort you have made and suceed to get this much big database. I was randomly searching for movie converter tools, to get information from the internet (like IMDb, Amazon Movie Databases) and come across your website. I really felt happy and you have done excellent job. If you have any idea how to get the movie information (title, cast, director, music etc etc) as well list of all songs (by singer). Thanks Jag
kannu71 19 Feb 2007 12:00 pm: really a nice piece of work, waiting for more
Sravya 19 Feb 2007 12:00 pm: Hello, We are not able to search the desired songs.You are not providing the search box.Sometimes the search box appears,sometimes not. We liked the website,but we want you to always provide the search box.
Dhimant Panchal 19 Feb 2007 12:00 pm: it is a nice program to find a song. but i have one problem. your program search the song but cannt play it. it sows an error message to contect the real network support center. you have any solution ? pls tell me. Thanks.
rahul kumar gupta 19 Feb 2007 12:00 pm: the site is so good but… if i want select all song how?
milton 19 Feb 2007 12:00 pm: What site give the fastest download in window media player? Please list this to me.Thanks.
Rahul 19 Feb 2007 12:00 pm: Excellent. Excellent. Is it possible to download songs on iPod. I am not an expert in technology and hold basic knowledge but would appreciate if somebody can tell me if it is possible to download songs from this site on my iPod. Thanks.
parag 19 Feb 2007 12:00 pm: gr8 effort man, even i too a music freak. I hv collected thousands of songs in mp3 format, but same problem here too, how to maintain database, for few i generated playlist & put on my site:- http://geocities.com/direct2parag/new/coolgoose/ still i found it cumbersome. will u plz help me out to maintain a good database for the same….. solicit ur co-operation…. Thanks. parag
priyanka 19 Feb 2007 12:00 pm: raaga
E.Srinivasarao 19 Feb 2007 12:00 pm: E X C E L L E N T
ASHOK 19 Feb 2007 12:00 pm: your job is ecellent can you upload the tamil karoake also? ashok
sen 19 Feb 2007 12:00 pm: u have done a commendable job - ‘anand’ a gaya
A K V K Rao 19 Feb 2007 12:00 pm: Excellent work. Brilliant Idea. Good luck. Rao.
Nitin Patel 19 Feb 2007 12:00 pm: A very good information. I would also like to suggest 2 different websites http://musicmirchi.net and http://dhintana.com
Mohd Abdullah 19 Feb 2007 12:00 pm: no comment
Binosh Nambiar 19 Feb 2007 12:00 pm: Mindblowing..Excellent……!!!!! Anand you have really done a great job… Can u provide URL Addresses of all songs????? Actually i have an Account in Zorpia.com… I want to creat my Own playlist..
Chetan 19 Feb 2007 12:00 pm: This is great. The concept, the implementation, the application of it. Everything is great ! I love it when ideas get implemented and get implemented in a way that they stay around e.g. your bi weekly updates is the best part.
prakash 19 Feb 2007 12:00 pm: i like you
AKASH 19 Feb 2007 12:00 pm: this is great site. i am enjoye it this concept. best of luck
Neha singh 19 Feb 2007 12:00 pm: i like old indian songs
Nitin 19 Feb 2007 12:00 pm: I also like old India songs just like Neha :)

MP3 Songs Downloads
sangye 19 Feb 2007 12:00 pm: Hi I really do hope you will write me back. i have seen your song collection and it is fantastic. I live in europe and teach bollywood dancing. so your site is perfect. thanks as the saying goes you are a genius for many. Ah but lets see if you will be one for me too. i have a hindi song it is a mix of east meets west. cannot find the name of the movie, everytime i tupe in 2 to 3 words i get the silsilay as the first words are same. so here goes. " oh oh oh, oh oh oh, oh oh oh feel it oh sooniya oh oh mahiya oh la la la … feel it feel it , meri chanii tu meri sona tu mera hera tu mera ,,,,.. zindagi ka guzara tere bin nahi hona ishaq hai tum se yaara tum ko nahi khona. )2) male - oh sooniya oh mahiya…. Waiting for your reply. thx dear
vaidehi 28 Aug 2008 12:45 pm: GGGGGGGGGGGGOOOOOOOOOOOOOOOOOOOOOOOOOOOD
murali 6 Nov 2008 8:55 pm: solla vaarthaigal illa………….

born genius.. www.cooldls.com
samarth 13 Dec 2008 2:01 pm: hi..i couldn’t open ur web page search engine..it becomes totally white page..and the web messege is also showing done..why is it so??i too want to take taste of your music search engine as i m one of great lovers of music..so plz show me way..
ratna 28 Dec 2008 8:20 am: anand you are simply briliant.my husband and I are both in music profession and it helps a lot to find a song.Great job. God bless you!
Vinoth 15 Jan 2009 11:26 pm: Mr. Anand

I am from chennai. for long i have been trying to do some thing like you have done. only that i started in 2009 and not in 2005. anyway now what i am looking for is to collect and prepare a complete database of old and new hindi songs. now my work has become simpler as you already have collected. i would further interact with you so if you give me your email it will be helpful to me.
Vishwanath.P 7 Feb 2009 6:56 am: You are really great “SIR”. You if, with one man army did all these things, my hats off to you.. Well i am also like you, in the sense that i wanted to a complete NEAT collection of songs… Thank you
Hemant Shah 21 Feb 2009 5:06 pm: Splendidly done - What marvellous use of available tools
Sriram 2 Mar 2009 11:32 pm: Good job..
ROHAN GANDHI 20 Mar 2009 6:58 am: where i can find tag songs or tag list for hindi songs
Srikanth 23 Jun 2009 7:06 pm: Excellent work! The way you have thought about the whole process is remarkable and the fact that you shared every step is highly appreciable. Helps a lot to see stuff like this for working on other projects. Thanks a lot!
Saravanan 15 Jan 2010 12:42 am: Simply Excellent, the thought process is simple great Fantastic job, clean and neat :)
NSaravanan 14 Dec 2010 7:16 am: Hi Anand, Simply superb. I am not very familiar with programming…I sometimes adapt few VB codes for my excel. that’s all. Is there a list of Tamil movies with tags, which you can share? Also, have you tried copying the Album Artwork for the movies / songs? cheers, Saravanan, Gurgaon
Akshay 27 Jun 2012 3:47 pm: Hey man. I am currently in the process of the tagging my hindi mp3 files as well and like you am a serious media monkey user. I was wondering if you could help me with 1) a comprehensive database of songs with id3 tags and 2) set up a way to auto tag my files with missing info. Thanks
Parthasarathy-82, @ Sarathy 1 Jun 2011 10:00 am: Dear Sir, You have achieved 30% of quantum., that means you have conquered Tamil Nadu completely many decades back. About 60% have finished becasue you have conquered entire South India and made Bollywood admired about your excellency, but your goodself is about to achieve 100% centum per cent by conquering the entire world by scoring different melodies in almost about all world languages in the remaining period. Why you know sir ….. Because there is no another ILAYARAJA SIR IS GOING TO BE BORN. This is plain truth and universal fact as well. “pudu raagam padaippadaale “NAANUM IRAIVANE” Yes, of your are the Iraivan of WORLD MUSIC OF 21ST CENTURY OF UNIVERSE Regards, Parthasarathy-TVK NAGAR - CHENNAI - 82, @ Sarathy
Nirmala 13 Apr 2011 9:06 am: It is very interesting to read your adventure. Music drives us crazy sometime. In turn wonders m ay happen.As I am also into your craze why cant you add features so that songs can be played remotely also voice input would be another challenge. What do you say. I have not yet tried your application. Happy music days !
Mohan Kethees 30 Aug 2011 2:30 pm: Hi. Nice work. 2 things.
1. How do i use the python crawler you have linked in the help.
2. Why not make it a web service? Idea for web service would be to query for a song and you get full ID3 detail or query for a movie and you get all the song ID3 details in that movie. It would help for I am willing to help But I am a C# developer and have little to no knowledge in python or Perl. Thank you
shiva prakash 17 Aug 2011 12:01 pm: Thanks a lot for your dedication
Ambarish 19 Jul 2011 7:07 pm: Thank you very Much…Awesome… I can’t believe i got all my favourite songs here in one single website…Thank you very much again…Good Luck
Tushar 21 Mar 2011 8:01 am: Hats off to you! You ave done excellent job. Hope the search site you have created will be always up. and thanks a lot. :)
nk 26 Jun 2013 10:47 am: Excellent work putting this up! Thank you. FYI, Musicindiaonline links don’t seem to work.
Nadeem 25 Aug 2015 9:02 am: [http://music.cooltoad.com/music/download.php] This site is down, is there any other link for those songs???

Programming theorems

Wed, 08 Nov 2006 12:00:00 +0000

Programming theorems.

The likelihood of Perl being involved in a system is directly proportional to the length of time the system has been in maintenance.

Every 5 minutes you spend writing code in a new language is more useful than 5 hours reading blog posts about how great the language is.

Think twice before presuming that CSV is a nice little easy file format. (see Leon)

Automated resume filtering

Tue, 03 Oct 2006 12:00:00 +0000

I had to screen resumes from a leading MBA school. I’m lazy, and there were hundreds of CVs. So after procrastinating until this morning, I decided on 2 principles:

I will not spend more than 45 minutes on this. (That’s the duration of my train ride to office.)
I will not read a single CV. (I would write a program.)

The CVs were in a single PDF file. I saved it as text (it shrunk from 66MB to 1.6MB without the photos). Then I wrote a Perl program to filter CVs by keywords. We were looking for people with an interest and/or experience in IT consulting, so I picked “technology”, “consulting”, “SAP”, “IBM”, “Accenture”, “Deloitte”, etc.

Anyone without these keywords would fall out of the list. This eliminated 75% of the crowd. But since I didn’t want to read the rest, I used my favourite text-analysis technique: concordance. I extracted 3 words on either side of each keywords, and just read those. It was easy to see who’d “worked with suppliers like IBM” as opposed to who’d worked at IBM.

That’s it! I managed to cut the list down to 10%. Better yet, I also had a preference ranking. People with multiple keywords ranked higher than those with fewer keywords. And all this took little more than my train ride to office.

I can see this going to the next level. It’s easy to write a customised rejection letter, depending on which keywords are missing for each person.

Now, if it’s this easy to filter resumes, I can see every organisation do it in a few years. Which means, you need to write resumes for machines as well, not just for humans! For example, on my next CV, I’ll make sure I include the words “Boston Consulting Group” as well as “BCG” – just in case the software searches for only one of those keywords. Further, I’ll make sure I avoid spelling mistakes!

Comments

Siddharth Nagpal 3 Oct 2006 9:17 pm: How rational it is to screen CVs, like this..instead if u cud hv glanced thru CVs..then taken a decision it wud hv been lot more intelligent..than writing a search program..
S Anand 4 Oct 2006 6:44 am: When I manually scan CVs, I miss out a lot of keywords that I want to spot – simply because humans are not good at scanning accurately. For example, when searching for SAP in the CVs, I missed it in half the CVs that the program had told me had SAP – and I initially thought my program was wrong. Contrawise, programs don’t look for synonyms or abbreviations. So I need to make sure the keywords are accurate – otherwise I miss out on potential candidates. So there’s flaws in both approaches, but they’re complementary. Meaning, what you miss in one, you catch out on another. Since another colleague is doing the manual scanning, my using a program helps, I think.
Kaviraj Nair 4 Oct 2006 9:31 am: but I am sure you wanted to spend more than 45 minutes to decide the fate of those students!! no offence!! what say you??
S Anand 4 Oct 2006 10:51 am: They haven’t started applying to us yet. We’re just fishing around for good candidates ourselves. So if it’s anyone’s loss, it’s ours. If anyone applies, the CV is read and that’s a separate process. This is, like I said, proactive fishing.
Also, I don’t feel bad about it being 45 minutes: if I’d run through the CVs manually, I’d have taken 3-4 hours at least for the same quality!
Sorry – guess I’m not being too considerate here :(
Sudheer 4 Oct 2006 4:46 pm: Hi Anand, Being an ex-placecommer from ur alma mater… I can assure you that students/prospects looking for a job are always a couple of steps ahead of the firms… Simply because students have much more to lose/gain than corporates…
Sudheer 4 Oct 2006 4:51 pm: What I am saying is that such programs/ algos might not yield most honest results. Even though I accept your effort - returns calculation - It just supports my argument that corporates have less to lose/gain in the recruitment process
S Anand 4 Oct 2006 5:09 pm: Sudheer, I agree: students have more to lose. I see resume-filtering software becoming more popular. I know Google uses it as a process. BCG automates much of its screening using Excel. I’m sure it’s going to be more popular. I do hope students stay ahead. I wonder what tecniques will help, though. I suspect very different strategies will be required for machines – sort-of like search engine optimisation.
Sai 4 Oct 2006 7:42 pm: You really have to be extensive in your ‘keyword’ list don’t you? And any use for fuzzy logic here? For example somebody who has package implementation experience can quickly ramp up an learn other packages? Same for analyst kind of jobs - technology is probably the least important of the factors - what matters is the vertical industry knowledge. My 2 cents!
S Anand 4 Oct 2006 9:01 pm: Sai, you’re right, the keywords have to be broad. I did use some pattern matching (for e.g, when looking for banking experience, using patterns like bank*; using synonyms like financ*, insur*; using abbreviations like FS, etc.) The things is, once I match and get snippets around the words, I can read the words and get the context. That’s practically the equivalent of my software telling me where to read on the CV. Then I make a judgement call. But you’re right about this example. I looked for SAP and ignored other packages. Will add a few more keywords and re-run it. Thanks!
ND 5 Oct 2006 5:35 am: Like your blog :-). I get “screened” resumes and 75% are irrelevant! For e.g. When I want Mercury ITG profiles, I get is Mercury QC profiles. I sift through “screened” resumes useing Edit-Find option in MS Word often using more verbs (both present and past tense) than nouns. I spend 10-15 minutes on identifying these “keywords”. MS Word takes me to the spot on the doc and I read around. I don’t get more than 50 “screened” resumes and all are .doc files. And I open upto 12 docs at a time without a system crash. Don’t mind some Type 1 error as long as Type 2 error is minimized!
S Anand 5 Oct 2006 5:59 am: I know what you mean, ND. And my guess is, this is how it’s done in many places. I can’t believe everyone reads through every word of every CV. Yet so many students indignantly object to this reality!
Sathya 5 Oct 2006 9:37 am: Perhaps a tip for resume writes … have a keyword section and put all the keywords out there -like the one u have in th “Contact” section. Putting all the keywords in the context of the CV might bloat up the size of CV.
S Anand 6 Oct 2006 6:18 am: That would work Sathya, but perhaps look a little awkward. Maybe a smarter thing to do is to prepare a keyword list, and ensure that’s woven into the text.
sathya 6 Oct 2006 12:42 pm: May be looks odd to the humans … but not so for machines :-) It could be made to look less awkward by making it a tag cloud. That would also indicate the comfort/expertise level (relatively). Is there a way to mark certain sections in word doc that appears when opened but does not get printed ??
Nitu Agarwal 6 Oct 2006 1:32 pm: Hi Anand, I read this article of your and have been trying to add key words which are generally asked in the job description.I hope to get some positive feedback now.I have been in London since last 4 months but not able to get a suitable job may be my CV didn’t have key words required in the UK.(Just to let u know i’m an Chartered Accountant from India)
Nitu 6 Oct 2006 1:37 pm: Hi Anand , I read this article of urs yesrterday and have been trying to add the key words as required by the job description.Hope there’s something positive soon.I have been in London for around 4 months and not been able to find a suitable job for myself ,may be my CV lacked key words.(Just to let u know i’m a Chartered accountant from India).It is nice reading ur articles.
S Anand 6 Oct 2006 1:48 pm: Sathya, yeah, I guess you could do that – a tag cloud would look funky, and you could always hide sections (maybe font colour?) It’d be interesting to see some CVs like that :-)
S Anand 6 Oct 2006 1:49 pm: Nitu, to my knowledge, using keywords is not that prevalent yet. But yeah, way to go!
Nitu 6 Oct 2006 2:22 pm: Not using the key words seperately but adding in the sentences to make sense and almost the same meaning as i was trying to tell earlier.
S Anand 6 Oct 2006 3:09 pm: Nitu, that’s probably the best thing. Happy to see at least one person have a better shot because of this article. Spread the word :-)
joey 28 Oct 2006 10:37 pm: anand Many recruitment firms use this method in Oz (Australia). It works - the right words get an phone interview call. For Nitu - my advise is to scan job ads and incorporate those sentences into the resume (nt just words) this will guarantee a higher response rate. worked for both my wife and me. And the first job in a new country is always hard.
Swapna 29 Mar 2007 8:17 am: Nitu, any improvement in reponse? I am also a CA from India and looking for job in UK!! Its seems really difficult to get an interview call here!
Georgie Girl 31 Oct 2009 4:36 am: A really interesting idea. In Canada I’ve seen the use of scanning program as well. I have been in HR and had to manually scan 100’s of resumes and I can tell you that you are cross-eyed at the end. I guess a resume should have a footer with tag line extras, what would a good set include do you think?

Software for my new laptop

Tue, 26 Sep 2006 12:00:00 +0000

And so, thanks to Infosys Consulting being spun off as a separate legal entity in the UK, I got my new laptop. (Because our old laptops were legally the assets of Infosys Technologies Ltd, and not Infosys Consulting Inc. Weirder things have happened, but who’s complaining?)

My old Toshiba Portege A200 has been replaced by a Dell Latitude D420 (which I was dreaming for, after having just read Jeff’s post on big laptops).

Firstly, it’s light. I thought my Toshiba was light compared to the Dell monsters others had, but this weighs 1.4 kgs! Secondly, it’s thin. It’s thinner than some of the paper notebooks I used to carry. It’s hard to imagine where we will be in 5-7 years time if innovations keep rolling in at this rate.

There were only two (minor) problems I saw with it. It didn’t have an S-Video port – so I can’t watch movies on TV. And it had a fairly small (12") screen. Being a wide screen, I get a lot less height than I used to. I’m still having some trouble getting used to that, especially when browsing tall pages but it’s a good laptop for playing games such as 올인구조대.

My weekend was like a kid in a candy store. Here’s what I did.

Uninstalled useless software: The laptop came with Roxio Easy CD Creator 5 Basic and PowerDVD 5.1. I got rid of them.

Copied all my files: I had about 25GB of data (15GB of music, 5 GB of books, 2 GB of video, 3GB of work). This was a bit tricky: some of my data was in SVN repositories, and I had to migrate them.

Configured the new system by literally running through each entry in the control panel, and ensuring that it’s the same as my old machine. Most of my changes are spartan (aimed at less eye-candy, usually). For example,

Display: I switch to Windows Classic and a black background. I used to do this because it takes less memory, but with 1GB of RAM, that’s no longer a consideration. I just got used to this. I also turn off all special effects, and remove everything except the Recycle Bin from my desktop. But the really useful thing is to turn on ClearType.
Taskbar and Start Menu: I switch to the Classic Start Menu, and turn off everything. Here’s what mine looks like now.
Toolbars: I like my toolbars to fit on one line. So I do some heavy customisation with the Internet Explorer toolbar to shrink it to a line. Similarly on the desktop toolbars.

Installed software. This is the fun part. I’ve made a number of changes to my software inventory.

ActivePerl: Perl is the only language I still know.
ActivePython: I’m learning Python. I’m not impressed, but the Python Image Library makes jigsaw quizzes easier than in Perl.
Apache HTTP server: To test this web page locally.
Audacity: To record and convert sounds. I’ve ditched Goldwave.
Camstudio: To capture screen sessions.
Crimson Editor: I’m still trying to pick a good text editor. (I need block operations, filtering, regular expressions and syntax highlighting. No editor seems to offer all of these. Filtering is especially rare.)
Cygwin: Just for the basic tools (head, tail, less). UnixUtils would do just fine, actually.
DX-Ball 2: The only game I play other than Solitaire, Minesweeper and Age of Kings.
Google Desktop: I can’t live without it any more.
Google Earth: I find Google Maps faster and better (because of the street maps).
Google Toolbar for IE: It practically replaces my address bar.
ImgBurn: I’m trying it instead of Nero. Except for multi-session support, it’s great.
Microsoft Reader: I have a lot of books in the .LIT format
Mozilla Firefox: Mainly for its extensions (Google Suggest, NextPlease, Tab Mix Plus, Google Bookmarks Button, Extended Statusbar, IE Tab, Live HTTP Headers, Web Developer)
Paint.NET: Quicker than Photoshop. Mostly I just crop images.
Picasa: Adequate for viewing photos.
Powertoys for Windows - TweakUI: To make Windows even more spartan.
Restoration: Can restore permanently deleted files.
Subversion: I like the version numbers (1, 2, 3, …) better than CVS’ (1.1, 1.1.1, 1.1.1.1, …).
VirtualDub: Mainly to convert videos from my digicam to DivX.
VLC player: When all else fails, I turn to VLC.
WinAmp: with Media Library import/export and Pepper
WinRAR: For the occasional .rar download

And finally, after reinstalling my SVN repository and copying my WinAmp playlists, Firefox bookmarks, etc, my new laptop feels as good as old.

Comments

Somnath 27 Sep 2006 4:03 am: How do you have admin rights our laptop? Most companies IT security policy means no admin rights on your machines (laptops or desktops).
S Anand 27 Sep 2006 7:08 am: Oh yes, thankfully! I can install pretty much anything, and the only issue is that these won’t be “supported” by our IT team. (But in reality, they’re actually thrilled to support these, when called :-)
Somnath 3 Oct 2006 3:30 am: One definitely good reason to work for Infy Consulting then ;-)
Michelle 5 Dec 2006 12:38 pm: Please tell me there is literature of non-fiction in 1977. Except for MOVIES!
Anand 8 Jan 2007 6:59 pm: Do you plan to embrace Windows Vista when it is released later this month ? Also I recommend the Firefox extension Colorful Tabs. It is pretty
S Anand 15 Jan 2007 8:16 pm: Guess I don’t have a choice about Vista. If Infosys upgrades, I upgrade.
Anonymous 3 Apr 2007 3:58 pm: I have also reently swicthed to D420. I can’t get it to record on audacity (or anything else). Previous Dell laptop was fine. Any suggestions
S Anand 3 Apr 2007 4:19 pm: I’m able to record using the Microphone on Audacity. But on my Toshiba, I was able to record what I was playing on the speaker as well, and that’s not possible on D420, thanks to the sound card. Also, the hard disk is incredibly slow.
Rockey Nebhwani 11 Jan 2010 4:02 pm: I would like to see the revised list of softwares on your laptop :-)
Software for my new laptop 2 | s-anand.net 27 Sep 2011 7:15 pm (pingback): […] for my new laptop 2 September 27th, 2011 Top 10 lists S Anand Time for a new laptop, and to replace software. Here’s my new […]
Software I currently use | s-anand.net 9 May 2014 6:23 pm (pingback): […] Every few years, I review the software I use. Here are some of my earlier lists. […]

Experiments in sound

Tue, 12 Sep 2006 12:00:00 +0000

Wikipedia says the human voice frequency for speech is between 85 to 155 Hz for men, and 165 to 255 Hz for women. That set me thinking.

What is the limit to our hearing?
How do sounds differ?
How can we synthesise speech?

What are the limits to our hearing?

Kids can hear frequencies from 20 Hz to 20 kHz, while adults hear only up to 12-14 kHz (Frequency Range of Human Hearing).

To check the lower frequency limit, I created an MP3 with sounds from 1 Hz to 100 Hz at 1 second intervals. Just play the sound, and see when you start hearing something. (Of course, whether you can hear something also depends on the volume of your speaker, the ambient noise, etc.) I could hear nothing for the first 40 seconds: so I can’t hear frequencies lower than 40 Hz.

PS: Don’t be worried if you don’t hear anything for a while. You’re not supposed to! Keep the volume at full level, though.

To check the upper frequency limit, I created this MP3 with sounds from 1 kHz to 20 kHz in 1 second intervals. Just play the sound, and see when you stop hearing anything. I couldn’t hear anything beyond 14 seconds: so I can’t hear frequencies beyond 14 kHz.

How do sounds differ?

I took this audio file of someone reciting vowels and plotted a spectrogram (below). A spectrogram plots time on the X axis and frequency on the Y-axis.

Some observations:

All the vowels have evenly spaced bars. (In this case, they’re all multiples of something around 120 Hz.)
‘u’ has the lowest frequency mix. ‘a’ spans from low to high. ‘i’ has a bit of low and a bit of high, nothing in the middle. ‘ai’ and ‘au’ look like ‘a’ followed by ‘i’ and ‘u’ respectively.

How can we synthesise speech?

I don’t know. There are lots of speech synthesizers. They sound robotic. I’m trying to see if knowing what sounds look like improves things. I’ll let you know if I do well.

Comments

Dhar 13 Sep 2006 5:09 am: How did you go about synthesizing the MP3s with increasing frequencies? Did you use some software for that? Similarly what software did you use to plot the spectrogram?
S Anand 13 Sep 2006 6:10 am: I wrote a perl script to generate the frequencies, and Spectrogram v.0.8 to create the spectrograms.
Sumit Dhar 16 Sep 2006 9:23 am: Dude, just downloaded the files. Is the sound stereophonic or monoaural? If it is mono, do you think creating an MP3 with stereophonic sound will make a difference? Cheers, D.
Sumit Dhar 16 Sep 2006 9:25 am: Loved the experiment. Lower: 42 Hz, Higher: 15 KHz
S Anand 16 Sep 2006 9:38 am: It’s mono. I doubt if stereo would make a difference. I’d be playing the same frequency on both speakers, which is what a mono MP3 does anyway.
Dhar 17 Sep 2006 2:05 am: The reason I asked about whether the sound is Mono or Stereo is because of my experience during eye tests. Some letters that were not visible when viewed with just one eye, were clearly readable when using both eyes. Was wondering if the same may be applicable in this case?
S Anand 17 Sep 2006 7:36 am: Ah, I see what you mean. Since both speakers would be playing the sound, hopefully this should not be a factor.

Python vs Perl

Wed, 11 Jan 2006 12:00:00 +0000

Python vs Perl. Sums up my feelings perfectly: Python may be better for larger projects, but for my meddling, I’ll stick to Perl. It’s served me well for 10 years.

Until 1999, I used Perl a fair bit, but no more than Java or C or anything else. My first “real-life” use of Perl was in 2000, when I was processing 600MB of IBES data. Access and SPSS couldn’t handle the load. Perl slurped all the data in a few seconds, though. A few years later, when processing bank data (3GB worth, this time), Perl again was the only saviour. In fact, between Excel and Perl (and CPAN), I think I have all the data analysis power I’ve ever needed. This blog, for instance, is written in an Excel spreadsheet, exported to XML, and converted into the blog format by Perl.

Matching misspelt Tamil movie names

Fri, 09 Dec 2005 12:00:00 +0000

I don’t like hunting for new songs either. Too much effort.

External recommendations like Raaga Top 10 help, but not much. I usually like only 1 of the top 10.

I don’t really know the recent music directors. But many interesting songs I’ve heard recently (like Ondra Renda in Kakka Kakka, Vaseegara in Minnale, and Kaadhalikkum in Chellame) are by Harris Jayaraj. So maybe if I can find the music directors I like, other songs by them would be good recommendations.

I have an automated way to find the music director for a movie. First, I spent a few hours renaming my MP3s to a Movie.Song.mp3 filename format (using Excel and Perl liberally). After that, I wrote a Perl program that reads movie names and the movie directors from Raaga and matches the Raaga movie names with my movie names. (Raaga has all but 5 movies whose songs I’ve heard.) Then I rate music directors based on my songs’ ratings.

Unfortunately, the matching worked only for 45% of my 273 movies. The rest were spelt differently on my list and Raaga. I checked CPAN if there was a way to match Tamil words roughly. The closest was Lingua::Phonology, but Jesse, the author, mailed me saying that was “like slicing your bread with a chainsaw”.

So I developed these rules. The -> arrow below is to be read as “is also spelt as”. By just applying them sequentially, I matched 33% more movies.

Vowel rules
AEdhiri -> Edhiri
kadhal kondEIn -> kadhal kondEn
chellamEY at end-> chellamE
sachIEn -> sachIn
marupadIUm -> marupadIYUm
OI, OY, OVI, OYI are all the same
AAthma -> Athma
azhagiya thEEye -> azhagiya thIye
abOOrva ragam -> abUrva ragam
Ignore H. It is redundant.

Consonant rules
arasakTCHi -> arasakSHi
CHippikkul muthu -> Sippikkul muthu
thenNDRal -> thenNRal
devar maHan -> devar maGan
bagaWathi -> bagaVathi
avvai shanmuGi -> avvai shanmuKi
konJi pesalam -> konCHi pesalam
anDha 7 naatkal -> anTha 7 naatkal
aBoorva sagodharargal -> aPoorva sagodharargal
agni natchaTHIRam -> agni natchaTHRam

The remaining movies either had spelling mistakes (e.g. Kilipethcu Ketkavaa) or had structural differences (Ilamai Oonjal Aadugiradhu vs Ilamai Oonjal Aadudhu). By permitting approximate matches using String::Approx, I was able to match 12% more, making my total accuracy ~90%.

Though this is good enough for identifying music directors, I’m working on improving the approximate matching rules. I hope to have 98% accuracy, and then I can match individual songs – and know who the singers are. Hopefully, this can be extended to other sites like MusicIndiaOnline, and who knows – maybe even IMDb.

Comments

Ashwin 10 Dec 2005 3:36 am: I use www.arthika.net for downloading high quality MP3’s. Their song ratings are good. This is my strategy.
S Anand 12 Dec 2005 6:55 pm: I tried it out… maybe it was just me, but I thought they had a smaller collection than Raaga. Is that right?
Ashwin 18 Dec 2005 6:41 am: Ya, They have only a small collection but he quality of the songs are very good.
Bala 9 Dec 2005 12:00 pm: Can you make the script you made downloadable for folks to normalize their unwieldy MP3 library? Even 80% would do for me considering the amount of chaos that prevails! :) Thanks

How social network analysis is used

Fri, 28 Oct 2005 12:00:00 +0000

An interview with the creators of InFlow (a software to map social networks) shows how social network analysis is used.

Comments

Dhar 30 Oct 2005 6:10 am: Anand, do you know of any freely available Social Network Analysis tool?
S Anand 31 Oct 2005 6:16 pm: Afraid not. I’ve only done this twice in projects, and I just used Excel and Perl both times.

Perl illiteracy

Thu, 15 May 2003 12:00:00 +0000

US high school seniors can’t read Perl. Not even a simple DeCSS program (obufuscated, of course).

Microsoft dumps Java

Wed, 12 Jul 2000 12:00:00 +0000

Microsoft dumps Java. Doesn’t worry me to much. I’ve never had much use for Java as a user. They’ve ‘adopted’ Perl, though. And I was always a Perl fan.

CGI.pm

Sun, 25 Jul 1999 12:00:00 +0000

I was playing around with CGI and wrote a script to determine when somebody last logged in. Read about CGI.pm.

perl on S Anand

Perl, 1994-2011

The Social Network

To Python from Perl

Comments

print level, residual, pattern, innResid

print count, pattern + (p,)

Caching pages on Apache

Comments

Launching applications

Comments

@setlocal enableextensions & python -x %~f0 %* & goto :EOF \

Load all the config file details in a hash \

load into a dictionary \

Reducing the server load

Comments

Splitting a sentence into words

Comments

Hindi songs online

Comments

Programming theorems

Automated resume filtering

Comments

Software for my new laptop

Comments

Experiments in sound

Comments

Python vs Perl

Matching misspelt Tamil movie names

Comments

How social network analysis is used

Comments

Perl illiteracy

Microsoft dumps Java

CGI.pm