How I do things

When you make a link, there’s no guarantee that the link will work 5 years later. Sites change their URL structure. I’m finding that many of my blog entries from 2000 are invalid.

Sometimes you want to link to a concept rather than a site. In such cases, it’s better to link to a Google query.

For example, rather than link to a site that defines SVG, I could link to the Google search define:SVG.

Rather than link to a tutorial on Excel array formulas, I could link to the Google search excel array formulas. I could even link to the first hit on Google for excel array formulas, mimicking the “I’m feeling lucky” button. This may change over time, but 5 years from now, it’ll still point to the most relevant link.

To link to the Google query for “excel array formulas”, just link to the URL http://www.google.com/search?q=excel+array+formulas. To link directly to the first result, add &btnI=I'm+Feeling+Lucky to the URL. (Linking to A9 is simpler: http://a9.com/excel+array+formulas)

PS: An alternative is to link to a permanent copy of the page from the Wayback machine (it has copies of my page all the way from May 2001 to Mar 2005). (You can’t use Google’s cache. When the site changes, the cache will soon change. But it’s a good defence against site downtime. Manually doing this is a lot of effort. Ideally, future browsers will automatically take you to the Wayback machine or the Google cache. (The Firefox plugins ErrorZilla and CacheIt come close.)

I’ve built a simple Sanskrit transliterator. You can type in words in English and it will spell them out in Sanskrit. You can copy-paste the Tamil above into Microsoft Word, etc.

Browse through my Javascript to see how it works. Feel free to reuse.

Calvin and Hobbes has some recurrent themes, like Hobbes pouncing, snow art, polls, letters to Santa, …

Over the last 5 years, I’ve transcribed the Calvin and Hobbes comics, and tagged them manually by theme. But can I generate themes automatically?

One way is to use Amazon’s statistically improbable phrases. It’s a list of words that occur a lot in a book, but rarely occur in others. It gives you a good feel of what topics the book is about.

Here’s how I did it:

1. Transcribe Calvin & Hobbes. This is 99% of the work.
2. Make a C&H word list. Just join all the words in Calvin and Hobbes. (Be careful about punctuation, and colloquialisms like “dunno”, “leggo”, etc.)
3. Get an English corpus. That is, get a big list of words in normally occurring text. I have some e-books, and I picked 23 megabytes worth of these as my corpus.
4. Compare the word frequency in C&H with the corpus. That is, compare the % of occurrences of a word in Calvin and Hobbes versus the corpus.
5. Display those with significantly higher frequency in C&H.

The list below has common Calvin & Hobbes words occurring 10 times as often as in normal text. It’s incredible how closely it relates to most of the themes.

(Big words occur more often. Dark words are more improbable.)

---

allowance assignment babe balloon bat bath beanie bedtime bee beep bet bike blaster boring bug bus butter calvin calvinball cartoon cent cereal cheat chew chocolate click comic cookie crunch dad dame derkins dictator-for-life dinosaur disgusting doll doomed dumb duplicate earthling explorer fang fearless ferocious flip flush frog frosted fun fuzzy genius goggle goodness goon grade gross grown-up gum hack hamburger hamster hate hero hideous hobbes homework huey insect invent jelly jerk jurassic kid leaf loot martian math mild-mannered mom monster moron motto munch mushy nickel oatmeal ouija pant peanut perspective pit playground poll porridge poster quiz recess rosalyn rotten rub sandwich santa scary sculpture scum shovel
sissy sitter sled slimy slug slushball sniff snow snowball snowman soak spaceman spiff splash spoil sport squirt steer sting stuffed stupendous sugar susie tickle tiger toy transmogrifier transmogrify tub tuna twinky tyrannosaur underwear vacation weird wham whiff worm wormwood

---

Summary: “Statistically improbable phrases” are a powerful tool for text analysis. You can apply it on any content and figure out what topics it talks about.

Update: Technically, these are “Statistically improbable WORDS”, not phrases. So I re-did this analysis using phrases instead of words.

I work across multiple computers (my office laptop, home laptop, client desktop) on a daily basis.

I used to transfer data across these by e-mailing them before I travelled. (I often forgot to do so.) Mostly, these are notes — like telephone numbers, things to buy, places to visit, etc.

Google Notebook solves the problem. But not entirely. I store a lot of my notes on spreadsheets, as lists. For example:

• Gadgets to buy (and accompanying research)
• Movies I’ve seen
• Books to read (and which library they’re available from)
• To do lists

That’s what I use Google Spreadsheets for — to share lists with myself, across computers.

Microsoft adCenter Labs has a demographics prediction engine. Based on a person’s search queries and web sites visited, it can predict their gender and age.

So I tried that on parts of the body, to see what men were interested in vs women.

topic	male	female
hair	25%	75%
eyes	33%	67%
cheek	33%	67%
hands	33%	67%
lips	36%	64%
ears	39%	61%
fingers	40%	60%
forehead	42%	58%
nose	43%	57%
neck	46%	54%
beard	55%	45%
moustache	58%	42%
leg	60%	40%
palm	61%	39%
toe	64%	36%

While I can understand men being more interested in beards and moustaches (perhaps even legs), why are they far more interested in toes than women?

Cut and paste has become easier. So we make less effort to understand. We don’t need to. Like when we pay less attention if we’re recording a lecture.

Solution? I suggest the Tunnel in the Sky strategy. Rod Walker is going for survival training on an alien planet, and asks his sister, Captain Walker…

“Uh, Sis, what sort of gun should I carry?”

“Huh? Why the deuce do you want a gun?”

“Why, for things I might run into of course.”

“Your only purpose is to stay alive. Not to be brave, not to fight. One time in a hundred a gun might save your life; the other ninety-nine it will tempt you into folly.”

“Did you take a gun on your solo test?”

“I did. And I lost it the first day. Which saved my life. I know how good a gun makes you feel. You’re ready for anything and hoping you’ll find it. Which is exactly what is dangerous about it – because you aren’t anything of that sort.”

So, don’t take a gun.

Don’t record lectures. Don’t give yourself the illusion of perfect memory.

Don’t bookmark for future reading. You won’t read it later.

Don’t cut and paste. You don’t understand it now. You won’t understand it later.

I was reading John Battelle’s The Search , and realised: We don’t sit down on the computer and say, “Let’s do a search”.

True. We want to get something done. We know it’s out there somewhere. We search.

So every search on a search engine is a commercial opportunity. Contrawise, every site must let people to do what they want to do on the site.

Think… What do people want to do when they’re on YOUR site?

On a related note, 60% of the search queries that lead to my site this year were Calvin and Hobbes quotes. “i can’t help but wonder what kind of desperate straits would drive a man to invent this thing.” topped the list (Calvin referring to a yo-yo), with i always catch these trick questions following closely.

People searching for Excel related stuff were next (20%): excel indirect(address(, row() excel offset address and the like.

A few were also looking for me by name or school (10%).

The last 10% ranged from the puzzling to the bizarre, including these gems.

• michalengelo hidden skull. Probably looking for the alleged hidden skull in Last Judgement.
• googlemail access between england and india. Why? Did he think there wouldn’t be any?
• greenwich meridien time for india. This is usually the same in Greenwich and in India. Rest of the world too.
• origin of monkey in fez. What?
• address of sexy girl in ahmedabad. But why my site?
• address of tool makers for converting html to xml in chennai. Do they have a license to convert?

On the IMDb top 250, you normally see a correlation between the number of votes and the rating for a movie. Better rated movies are more watched. The outliers are interesting.

The movies that are popular despite not having a high rating are:

• The Matrix
• The Sixth Sense
• Gladiator
• Star Wars 3: Revenge of the Sith
• Pirates of the Caribbean

I can understand why The Sixth Sense, Pirates of the Caribbean and especially The Matrix are on this list — geeks would have watched these and voted on IMDb, though their voting need not have been high. But why are Gladiator and Sixth Sense on that list?

Movies that are highly rated, but not as popular are:

• The Godfather 1
• The Godfather 2
• Seven Samurai
• Rear Window
• The Good, The Bad, The Ugly

Seven Samurai and The Good, The Bad, The Ugly probably didn’t get the votes they deserve because they’re written in their Japanese and Mexican names on IMDb. I hadn’t seen them for a long time for the same reason. As for The Godfather, I personally think it’s just overrated. But Rear Window? That’s a surprise. Hitchcock thriller with all the classic elements…

Another correlation is between the rating and the year of the movie. Early movies get lower ratings than recent movies. Technique could be the reason, but I doubt it. In any case, some movies stand out of their time.

• The Shawshank Redemption
• The Godfather 1
• The Godfather 2
• Seven Samurai
• Rear Window
• Casablanca
• Citizen Kane
• Metropolis
• M
• Modern Times
• City Lights

I haven’t seen Metropolis or M. But among the others, I think Citizen Kane is the one that deserves to stand out, if only for portraying the anti-hero, and for not having a happy ending. The Shawshank Redemption was a bit of a surprise. Few people that I know have heard of it. And yet, there it is, right on top.