statistically-improbable-phrases on S Anand

Statistically improbable phrases on Google AppEngine update

Sat, 12 Apr 2008 12:00:00 +0000

I’ve added some interactivity to the Statistically improbable phrases application. You can now:

Filter out stopwords
Dynamically filter infrequent words and commonly used words
Dynamically play with the contrast and font size

Comments

Srikanth 12 Apr 2008 12:00 pm: Dear sir, I was searching for Ilayaraja songs and came across your wonderful compilation of 15 wonderful articles. Good one. Please do write more on music.
Collin 12 Apr 2008 12:00 pm: I love this application. Because now, I can create a url to NY Times, and see what is the main subject of the day. :)
S Anand 12 Apr 2008 12:00 pm: Thanks, Colin!
Spencer 12 Apr 2008 12:00 pm: I was curious as to whether or not I could use this pointed into a specific personal corpus to separate documents from one another.

Statistically improbable phrases on Google AppEngine

Tue, 08 Apr 2008 12:00:00 +0000

I read about Google AppEngine early this morning, and applied for an invite. Google’s issuing beta invites to the first 10,000 users. I was pretty convinced I wasn’t among those, but turns out I was lucky.

AppEngine lets you write web apps that Google hosts. People have been highlighting that it give you access to the Google File System and BigTable for the first time. But to me, that isn’t a big deal. (I’m not too worried about reliability, and MySQL / flat files work perfectly well for me as a data store.)

What’s more interesting unlike Amazon’s EC2 and S3, this is free up to a certain quota. And you get a fair bit of processing power and bandwidth for free. One of the reasons I’ve held back on creating some apps was simply because it would take away too much bandwidth / CPU cycles from my site. (I’ve had this problem before.) Google quota is 10 GB of bandwidth per day (which is about 30 times what my site uses). And this is on Google’s incredibly fast servers It also offers 200 million megacycles a day. That’s like a dedicated 2.3 GHz processor (200 million megacycles = 200,000 GHz x 1 second ~ 2.3 GHz x 86,400 seconds/day) – better, because this is the average capacity, not peak capacity. The only restriction that really worries me is that only 3 apps are allowed per developer.

So I decided to give a shot at publishing some code I’d kept in reserve for a long time. You may remember my statistical analysis of Calvin & Hobbes. For this, I’d created a script in Perl that could generate Statistically Improbable Phrases (SIPs) for any text. This is based on (a somewhat limited) 23MB corpus of ebooks that I had. I’d wanted to put that up on my website, but …

AppEngine only uses Python. So the first task was to get Python, and then to learn Python. The only saving grace was that I was just cutting-and-pasting most of the time. Google wasn’t helping:

Anyway, the site is up. You can view it at sip.s-anand.net for now. Just type a URL, and it’ll tell you the improbable words in that site.

Visit sip.s-anand.net

Technical notes

I realise that these are statistically improbable words, not phrases. I’ll get to the phrases in a while.

The logic is simple:

Get the frequency of words in a corpus. I pre-generated this file. It has over 100,000 words.
Get the URL as text. Rather than muck around with Python, I decided to use the W3 html2txt service.
Convert the text to words. Splitting text into words is tricky. For now, I’m simply assuming that any group of letters is a word, and anything that’s not a letter is a word delimiter.
Find the relative frequency (improbability) of words. This is the frequency in the URL divided by the frequency in the corpus, normalised (i.e. scale it so that the maximum value is 1.0).
Create a tag cloud. I use the word frequency as the size and the improbability as the colour. You need a bit of mathematical jugglery to get the pattern right. Right now, I’m taking the 6th root of the improbability and the logarithm of the frequency to get a reasonably smooth tag cloud.

The source code is at statistically-improbable-phrases.googlecode.com.

Update: 12-Apr-2008. I’ve added some interactivity. You can play with the contrast and font size, the filter out common or infrequent words.

Update: 22-Apr-2008. Added concordance. You can click on a word and see the context in which it appears.

Comments

Sasidhar 8 Apr 2008 12:00 pm: Interesting. Great to see you’ve already developed an App. :) I was trying it out, and looks like something broke. Here’s the error. Traceback (most recent call last): File “/base/python_lib/versions/1/google/appengine/ext/webapp/__init__.py”, line 486, in __call__ handler.post(*groups) File “/base/data/home/apps/sip/1.35/sip.py”, line 64, in post result = urlfetch.fetch(“http://cgi.w3.org/cgi-bin/html2txt?noinlinerefs=on&nonums=on&url=" + urllib.quote(url)) File “/base/python_lib/versions/1/google/appengine/api/urlfetch.py”, line 95, in fetch raise DownloadError() DownloadError
Glenn 8 Apr 2008 12:00 pm: This is a great app and uses some techniques I wanted to learn more about. The source code seems out of date with the currently running app. Any chance you can update the source .zip file as well? Thanks in advance.
Brandon 8 Jun 2011 10:40 pm: The app is really, really neat. Is there any chance there will be a version that can do the same for Word files?

Statistically improbable phrases

Wed, 23 Aug 2006 12:00:00 +0000

Calvin and Hobbes has some recurrent themes, like Hobbes pouncing, snow art, polls, letters to Santa, …

Over the last 5 years, I’ve transcribed the Calvin and Hobbes comics, and tagged them manually by theme. But can I generate themes automatically?

One way is to use Amazon’s statistically improbable phrases. It’s a list of words that occur a lot in a book, but rarely occur in others. It gives you a good feel of what topics the book is about.

Here’s how I did it:

Transcribe Calvin & Hobbes. This is 99% of the work.
Make a C&H word list. Just join all the words in Calvin and Hobbes. (Be careful about punctuation, and colloquialisms like “dunno”, “leggo”, etc.)
Get an English corpus. That is, get a big list of words in normally occurring text. I have some e-books, and I picked 23 megabytes worth of these as my corpus.
Compare the word frequency in C&H with the corpus. That is, compare the % of occurrences of a word in Calvin and Hobbes versus the corpus.
Display those with significantly higher frequency in C&H.

The list below has common Calvin & Hobbes words occurring 10 times as often as in normal text. It’s incredible how closely it relates to most of the themes.

(Big words occur more often. Dark words are more improbable.)

allowance assignment babe balloon bat bath beanie bedtime bee beep bet bike blaster boring bug bus butter calvin calvinball cartoon cent cereal cheat chew chocolate click comic cookie crunch dad dame derkins dictator-for-life dinosaur disgusting doll doomed dumb duplicate earthling explorer fang fearless ferocious flip flush frog frosted fun fuzzy genius goggle goodness goon grade gross grown-up gum hack hamburger hamster hate hero hideous hobbes homework huey insect invent jelly jerk jurassic kid leaf loot martian math mild-mannered mom monster moron motto munch mushy nickel oatmeal ouija pant peanut perspective pit playground poll porridge poster quiz recess rosalyn rotten rub sandwich santa scary sculpture scum shovel sissy sitter sled slimy slug slushball sniff snow snowball snowman soak spaceman spiff splash spoil sport squirt steer sting stuffed stupendous sugar susie tickle tiger toy transmogrifier transmogrify tub tuna twinky tyrannosaur underwear vacation weird wham whiff worm wormwood

Summary: “Statistically improbable phrases” are a powerful tool for text analysis. You can apply it on any content and figure out what topics it talks about.

Update: Technically, these are “Statistically improbable WORDS”, not phrases. So I re-did this analysis using phrases instead of words.

Comments

DMac 23 Aug 2006 12:00 pm: This is great. Can you be more specific on how you did this? You say that the words listed are “common” words that appear more than 10 times more often - what is your criterion for “common”? Also, what were the ranges of improbability and oftenness that you mapped into the size and color of the results. How are improbability and oftenness different, anyway? Finally, how did you handle any words in C&H that didn’’t appear in your corpus? I’’m very interested in hearing more from you about how you did this - I’’m looking forward to hearing from you. Best regards…
Mugen 15 Sep 2010 4:33 pm: Any further directions to how exactly you came up with this/whether you used any software would be most helpful. Please tell us more about this. This is going to be awesome to help decide what to read. Thanks a load. Also, does amazon have this feature for most books? I browsed a lot of books in amazon but was only able to find this in one so far.