text-analysis on S Anand

Thu, 20 Nov 2025 00:00:00 +0000

While meditating, I realized 75% of “LULL” is the letter “L”. (This sort of thing happens a lot when I meditate.)

MUMMY (60% M) and DADDY (60% D) have lower percentage, but are longer, so maybe get a bonus?

I asked Claude Code what would top such a list. It picked a dictionary, generated the 333 words with 4+ letters and >50% concentration.

What did I like best? “ASSESSES”. 5/8 letters are “S”. That’s nearly two-thirds.

Then it wrote an xkcd-style story! Here are snippets:

I wrote a Python script to analyze 370,105 English words. Then I questioned my life choices.

*Does “mmmm” count? Look, I didn’t make the dictionary. I just downloaded it from the internet.

Notice something? After length 6, it’s just S-words and E-words all the way down. (It’s the grammatical duct tape of English)

I didn’t analyze Welsh, but I’m guessing their L’s would be extremely competitive.

The story way more fun than the analysis!

Check it out: https://github.com/sanand0/research/tree/main/repeated-letter-words

Learning: Find good styles to copy from. Here’s my style catalog: https://github.com/sanand0/prompts/blob/main/styles.md

Statistically improbable phrases on Google AppEngine update

Sat, 12 Apr 2008 12:00:00 +0000

I’ve added some interactivity to the Statistically improbable phrases application. You can now:

Filter out stopwords
Dynamically filter infrequent words and commonly used words
Dynamically play with the contrast and font size

Comments

Srikanth 12 Apr 2008 12:00 pm: Dear sir, I was searching for Ilayaraja songs and came across your wonderful compilation of 15 wonderful articles. Good one. Please do write more on music.
Collin 12 Apr 2008 12:00 pm: I love this application. Because now, I can create a url to NY Times, and see what is the main subject of the day. :)
S Anand 12 Apr 2008 12:00 pm: Thanks, Colin!
Spencer 12 Apr 2008 12:00 pm: I was curious as to whether or not I could use this pointed into a specific personal corpus to separate documents from one another.

Statistically improbable phrases 2

Wed, 14 Feb 2007 12:00:00 +0000

My earlier list of statistically improbable phrases in Calvin and Hobbes is technically just a list of “Statistically Improbable Words”. I re-did the same analysis using phrases. Here are the top 20 statistically improbable phrases (2 - 4 words only):

baby sitter chocolate frosted sugar bombs comic books doing homework fearless spaceman spiff() good night hamster huey ice cream miss wormwood new year peanut butter really think slimy girls spaceman spiff stuffed tiger stupendous man sugar bombs susie derkins watch tv water balloon

That is, these are the 2-4 word phrases whose frequency in Calvin and Hobbes is substantially (at least 5 times) higher than in the other books I have.

While doing this, the single biggest problem that stumped me was: what is a word?

Is “it’s” one word or two words?
Is “six-year-old” one word or three words?
How do I distinguish between abbreviations (g.r.o.s.s.) and full-stops without a space ( … homework.what’s a …)?
Does a comma always split words? (It doesn’t in numbers, like “3,500”)

The other problem is, phrases with more words are more improbable. Right now, if a phrase occurs 5 times more frequently in Calvin and Hobbes than my other books, I include it. But three-letter words rarely occur that often, and four-letter words even less so. Maybe I should have a lower cutoff for longer phrases.

Anyway, this analysis is a crude first approximation. Clearly Amazon’s gotten much further with their system.

Comments

satish 20 Feb 2007 3:56 am: Hey Stud, Satish here, your junior from IIMB. Trying to get in touch with you. Do mail me at satishkgv@hcl.in and let us get in touch.
Oblio 14 Feb 2007 12:00 pm: Fantastic job man! You have unlimited patience!
Reinhard Ebner 14 Feb 2007 12:00 pm: Hey, only just now came across your page, but of the hundreds, if not thousands of C&H sites and tools, this is the most useful I’’ve seen! R
juergwachter 14 Feb 2007 12:00 pm: hello
nice stuff.I intend splitting a text into single words. can you please give me a hint how to do this? I guess there are simple programs doing this.
Many thanks
joe 11 Aug 2009 2:01 am: Nice. Do you have a page where I can try out v2 (phrases)?
The Calvin and Hobbes search Takedown | s-anand.net 21 May 2010 11:53 am (pingback): […] was able to do a lot of cool stuff with this, like statistically improbable phrases and many amusing […]

Automated resume filtering

Tue, 03 Oct 2006 12:00:00 +0000

I had to screen resumes from a leading MBA school. I’m lazy, and there were hundreds of CVs. So after procrastinating until this morning, I decided on 2 principles:

I will not spend more than 45 minutes on this. (That’s the duration of my train ride to office.)
I will not read a single CV. (I would write a program.)

The CVs were in a single PDF file. I saved it as text (it shrunk from 66MB to 1.6MB without the photos). Then I wrote a Perl program to filter CVs by keywords. We were looking for people with an interest and/or experience in IT consulting, so I picked “technology”, “consulting”, “SAP”, “IBM”, “Accenture”, “Deloitte”, etc.

Anyone without these keywords would fall out of the list. This eliminated 75% of the crowd. But since I didn’t want to read the rest, I used my favourite text-analysis technique: concordance. I extracted 3 words on either side of each keywords, and just read those. It was easy to see who’d “worked with suppliers like IBM” as opposed to who’d worked at IBM.

That’s it! I managed to cut the list down to 10%. Better yet, I also had a preference ranking. People with multiple keywords ranked higher than those with fewer keywords. And all this took little more than my train ride to office.

I can see this going to the next level. It’s easy to write a customised rejection letter, depending on which keywords are missing for each person.

Now, if it’s this easy to filter resumes, I can see every organisation do it in a few years. Which means, you need to write resumes for machines as well, not just for humans! For example, on my next CV, I’ll make sure I include the words “Boston Consulting Group” as well as “BCG” – just in case the software searches for only one of those keywords. Further, I’ll make sure I avoid spelling mistakes!

Comments

Siddharth Nagpal 3 Oct 2006 9:17 pm: How rational it is to screen CVs, like this..instead if u cud hv glanced thru CVs..then taken a decision it wud hv been lot more intelligent..than writing a search program..
S Anand 4 Oct 2006 6:44 am: When I manually scan CVs, I miss out a lot of keywords that I want to spot – simply because humans are not good at scanning accurately. For example, when searching for SAP in the CVs, I missed it in half the CVs that the program had told me had SAP – and I initially thought my program was wrong. Contrawise, programs don’t look for synonyms or abbreviations. So I need to make sure the keywords are accurate – otherwise I miss out on potential candidates. So there’s flaws in both approaches, but they’re complementary. Meaning, what you miss in one, you catch out on another. Since another colleague is doing the manual scanning, my using a program helps, I think.
Kaviraj Nair 4 Oct 2006 9:31 am: but I am sure you wanted to spend more than 45 minutes to decide the fate of those students!! no offence!! what say you??
S Anand 4 Oct 2006 10:51 am: They haven’t started applying to us yet. We’re just fishing around for good candidates ourselves. So if it’s anyone’s loss, it’s ours. If anyone applies, the CV is read and that’s a separate process. This is, like I said, proactive fishing.
Also, I don’t feel bad about it being 45 minutes: if I’d run through the CVs manually, I’d have taken 3-4 hours at least for the same quality!
Sorry – guess I’m not being too considerate here :(
Sudheer 4 Oct 2006 4:46 pm: Hi Anand, Being an ex-placecommer from ur alma mater… I can assure you that students/prospects looking for a job are always a couple of steps ahead of the firms… Simply because students have much more to lose/gain than corporates…
Sudheer 4 Oct 2006 4:51 pm: What I am saying is that such programs/ algos might not yield most honest results. Even though I accept your effort - returns calculation - It just supports my argument that corporates have less to lose/gain in the recruitment process
S Anand 4 Oct 2006 5:09 pm: Sudheer, I agree: students have more to lose. I see resume-filtering software becoming more popular. I know Google uses it as a process. BCG automates much of its screening using Excel. I’m sure it’s going to be more popular. I do hope students stay ahead. I wonder what tecniques will help, though. I suspect very different strategies will be required for machines – sort-of like search engine optimisation.
Sai 4 Oct 2006 7:42 pm: You really have to be extensive in your ‘keyword’ list don’t you? And any use for fuzzy logic here? For example somebody who has package implementation experience can quickly ramp up an learn other packages? Same for analyst kind of jobs - technology is probably the least important of the factors - what matters is the vertical industry knowledge. My 2 cents!
S Anand 4 Oct 2006 9:01 pm: Sai, you’re right, the keywords have to be broad. I did use some pattern matching (for e.g, when looking for banking experience, using patterns like bank*; using synonyms like financ*, insur*; using abbreviations like FS, etc.) The things is, once I match and get snippets around the words, I can read the words and get the context. That’s practically the equivalent of my software telling me where to read on the CV. Then I make a judgement call. But you’re right about this example. I looked for SAP and ignored other packages. Will add a few more keywords and re-run it. Thanks!
ND 5 Oct 2006 5:35 am: Like your blog :-). I get “screened” resumes and 75% are irrelevant! For e.g. When I want Mercury ITG profiles, I get is Mercury QC profiles. I sift through “screened” resumes useing Edit-Find option in MS Word often using more verbs (both present and past tense) than nouns. I spend 10-15 minutes on identifying these “keywords”. MS Word takes me to the spot on the doc and I read around. I don’t get more than 50 “screened” resumes and all are .doc files. And I open upto 12 docs at a time without a system crash. Don’t mind some Type 1 error as long as Type 2 error is minimized!
S Anand 5 Oct 2006 5:59 am: I know what you mean, ND. And my guess is, this is how it’s done in many places. I can’t believe everyone reads through every word of every CV. Yet so many students indignantly object to this reality!
Sathya 5 Oct 2006 9:37 am: Perhaps a tip for resume writes … have a keyword section and put all the keywords out there -like the one u have in th “Contact” section. Putting all the keywords in the context of the CV might bloat up the size of CV.
S Anand 6 Oct 2006 6:18 am: That would work Sathya, but perhaps look a little awkward. Maybe a smarter thing to do is to prepare a keyword list, and ensure that’s woven into the text.
sathya 6 Oct 2006 12:42 pm: May be looks odd to the humans … but not so for machines :-) It could be made to look less awkward by making it a tag cloud. That would also indicate the comfort/expertise level (relatively). Is there a way to mark certain sections in word doc that appears when opened but does not get printed ??
Nitu Agarwal 6 Oct 2006 1:32 pm: Hi Anand, I read this article of your and have been trying to add key words which are generally asked in the job description.I hope to get some positive feedback now.I have been in London since last 4 months but not able to get a suitable job may be my CV didn’t have key words required in the UK.(Just to let u know i’m an Chartered Accountant from India)
Nitu 6 Oct 2006 1:37 pm: Hi Anand , I read this article of urs yesrterday and have been trying to add the key words as required by the job description.Hope there’s something positive soon.I have been in London for around 4 months and not been able to find a suitable job for myself ,may be my CV lacked key words.(Just to let u know i’m a Chartered accountant from India).It is nice reading ur articles.
S Anand 6 Oct 2006 1:48 pm: Sathya, yeah, I guess you could do that – a tag cloud would look funky, and you could always hide sections (maybe font colour?) It’d be interesting to see some CVs like that :-)
S Anand 6 Oct 2006 1:49 pm: Nitu, to my knowledge, using keywords is not that prevalent yet. But yeah, way to go!
Nitu 6 Oct 2006 2:22 pm: Not using the key words seperately but adding in the sentences to make sense and almost the same meaning as i was trying to tell earlier.
S Anand 6 Oct 2006 3:09 pm: Nitu, that’s probably the best thing. Happy to see at least one person have a better shot because of this article. Spread the word :-)
joey 28 Oct 2006 10:37 pm: anand Many recruitment firms use this method in Oz (Australia). It works - the right words get an phone interview call. For Nitu - my advise is to scan job ads and incorporate those sentences into the resume (nt just words) this will guarantee a higher response rate. worked for both my wife and me. And the first job in a new country is always hard.
Swapna 29 Mar 2007 8:17 am: Nitu, any improvement in reponse? I am also a CA from India and looking for job in UK!! Its seems really difficult to get an interview call here!
Georgie Girl 31 Oct 2009 4:36 am: A really interesting idea. In Canada I’ve seen the use of scanning program as well. I have been in HR and had to manually scan 100’s of resumes and I can tell you that you are cross-eyed at the end. I guess a resume should have a footer with tag line extras, what would a good set include do you think?

Statistically improbable phrases

Wed, 23 Aug 2006 12:00:00 +0000

Calvin and Hobbes has some recurrent themes, like Hobbes pouncing, snow art, polls, letters to Santa, …

Over the last 5 years, I’ve transcribed the Calvin and Hobbes comics, and tagged them manually by theme. But can I generate themes automatically?

One way is to use Amazon’s statistically improbable phrases. It’s a list of words that occur a lot in a book, but rarely occur in others. It gives you a good feel of what topics the book is about.

Here’s how I did it:

Transcribe Calvin & Hobbes. This is 99% of the work.
Make a C&H word list. Just join all the words in Calvin and Hobbes. (Be careful about punctuation, and colloquialisms like “dunno”, “leggo”, etc.)
Get an English corpus. That is, get a big list of words in normally occurring text. I have some e-books, and I picked 23 megabytes worth of these as my corpus.
Compare the word frequency in C&H with the corpus. That is, compare the % of occurrences of a word in Calvin and Hobbes versus the corpus.
Display those with significantly higher frequency in C&H.

The list below has common Calvin & Hobbes words occurring 10 times as often as in normal text. It’s incredible how closely it relates to most of the themes.

(Big words occur more often. Dark words are more improbable.)

allowance assignment babe balloon bat bath beanie bedtime bee beep bet bike blaster boring bug bus butter calvin calvinball cartoon cent cereal cheat chew chocolate click comic cookie crunch dad dame derkins dictator-for-life dinosaur disgusting doll doomed dumb duplicate earthling explorer fang fearless ferocious flip flush frog frosted fun fuzzy genius goggle goodness goon grade gross grown-up gum hack hamburger hamster hate hero hideous hobbes homework huey insect invent jelly jerk jurassic kid leaf loot martian math mild-mannered mom monster moron motto munch mushy nickel oatmeal ouija pant peanut perspective pit playground poll porridge poster quiz recess rosalyn rotten rub sandwich santa scary sculpture scum shovel sissy sitter sled slimy slug slushball sniff snow snowball snowman soak spaceman spiff splash spoil sport squirt steer sting stuffed stupendous sugar susie tickle tiger toy transmogrifier transmogrify tub tuna twinky tyrannosaur underwear vacation weird wham whiff worm wormwood

Summary: “Statistically improbable phrases” are a powerful tool for text analysis. You can apply it on any content and figure out what topics it talks about.

Update: Technically, these are “Statistically improbable WORDS”, not phrases. So I re-did this analysis using phrases instead of words.

Comments

DMac 23 Aug 2006 12:00 pm: This is great. Can you be more specific on how you did this? You say that the words listed are “common” words that appear more than 10 times more often - what is your criterion for “common”? Also, what were the ranges of improbability and oftenness that you mapped into the size and color of the results. How are improbability and oftenness different, anyway? Finally, how did you handle any words in C&H that didn’’t appear in your corpus? I’’m very interested in hearing more from you about how you did this - I’’m looking forward to hearing from you. Best regards…
Mugen 15 Sep 2010 4:33 pm: Any further directions to how exactly you came up with this/whether you used any software would be most helpful. Please tell us more about this. This is going to be awesome to help decide what to read. Thanks a load. Also, does amazon have this feature for most books? I browsed a lot of books in amazon but was only able to find this in one so far.

How to read when time is short

Sat, 12 Nov 2005 12:00:00 +0000

How to read when time is short. Read the “How To Find The Essential 20%” section carefully. Another interesting post from Bert on How to Learn More With No Extra Effort uses the principle in the post below to suggest we take a lot of breaks while learning.

i cdnuolt blveiee taht i cluod aulaclty uesdnatnrd waht i was rdanieg. the phaonmneal pweor of the hmuan mind is amazanig. aoccdrnig to a rscheearch taem at cmabrigde uinervtisy, it deosnt mttaer in waht oredr the ltteers in a wrod are, the olny iprmoatnt tihng is taht the frist and lsat ltteer be in the rghit pclae. the rset can be a taotl mses and you can sitll raed it wouthit a porbelm. amazanig huh? yaeh and yuo awlyas thohgut slpeling was ipmorantt.

Comments

Dhar 14 Nov 2005 9:18 am: Check out Google Analytics
Jayant 14 Nov 2005 10:00 am: And just hope the client does not know the remaining 80% or isn
Raj 15 Nov 2005 8:16 pm: what is the run?
Anonymous 16 Nov 2005 5:14 pm: i am confused with your email id is it s-anand@yahoo.com or s_anand@yahoo.com ?
S Anand 16 Nov 2005 6:50 pm: root underscore node at yahoo dot com, actually.
leke 17 Nov 2005 11:43 am: why can
S Anand 17 Nov 2005 3:34 pm: Sorry, apostrophes were causing problems in the comments. I’ve fixed them.
Sri 13 Dec 2005 11:00 am: This is interesting: http://www.mrc-cbu.cam.ac.uk/~mattd/Cmabrigde/

Amazon concordance

Fri, 22 Jul 2005 12:00:00 +0000

Amazon has a new feature called concordance that shows the frequently used words in a book like a tag list.

Textarc

Wed, 15 May 2002 12:00:00 +0000

Textarc – a visual way of representing text.

lorum ipsum dolor

Thu, 13 Dec 2001 12:00:00 +0000

Finally figured out that lorum ipsum dolor… means nothing.