natural-language-processing on S Anand

Longest repeated paragraph on Wikipedia

Tue, 26 May 2026 22:20:17 +0800

What is the most frequently occurring sentence in Wikipedia? ANS: A 213-word paragraph about how minor planets are named, which appears in 418 Wikipedia articles, word-for-word!

There are ~380,000 asteroids. Wikipedia has 418 pages for these - including one for each thousand-range of asteroids.

Every single one of these pages includes the phrase:

As minor planet discoveries are confirmed, they are given a permanent number by the IAU’s Minor Planet Center (MPC), and the discoverers can then submit names for them, following the IAU’s naming conventions. The list below concerns those minor planets in the specified number-range that have received names, and explains the meanings of those names.

Official naming citations of newly named small Solar System bodies are approved and published in a bulletin by IAU’s Working Group for Small Bodies Nomenclature (WGSBN).[1] Before May 2021, citations were published in MPC’s Minor Planet Circulars for many decades.[2] Recent citations can also be found on the JPL Small-Body Database (SBDB).[3] Until his death in 2016, German astronomer Lutz D. Schmadel compiled these citations into the Dictionary of Minor Planet Names (DMP) and regularly updated the collection.[4][5]

Based on Paul Herget’s The Names of the Minor Planets,[6] Schmadel also researched the unclear origin of numerous asteroids, most of which had been named prior to World War II.

Check out these pages | 85001-86000 | 213001-214000 | 269001-270000 | 380001-381000

This is not the only such common sentence. There are several more.

Here’s the Slovakia census note: 81 words that appear across 2,920 Wikipedia pages, like Sabinov District, Smolenice, Ilija, Slovakia, Baloň, … and thousands more!

Note on population: The difference between the population numbers above and in the census (here and below) is that the population numbers above are mostly made up of permanent residents, etc.; and the census should indicate the place where people actually mainly live. For example, a student is a citizen of a village because they have permanent residence there (they lived there as a child and has parents), but most of the time he studies at a university in the city

Note: As of 26 May 2026, this has been shortened to:

Note on population: The difference values of population numbers in the table “Population statistic” and in the sections “Ethnicity” & “Religion” is caused by the use of various statistical methods.

There are several more such that you can read about in The Paragraph That Appears 418 Times.

That also includes how Codex analyzed the Wikipedia structured dataset on Hugging Face and what else you can do with the data.

Shortening sentences

Wed, 10 Nov 2010 08:16:37 +0000

When writing Mixamail, I wanted tweets automatically shortened to 140 characters – but in the most readable manner.

Some steps are obvious. Removing redundant spaces, for example. And URL shortening. I use bit.ly because it has an API. I’ll switch to Goo.gl, once theirs is out.

I tried a few more strategies:

Replace words with short forms. “u” for “you”, “&” for and, etc.
Remove articles – a, an, the
Remove optional punctuation – comma, semicolon, colon and quotes, in particular
Replace “one” with “1”, “to” or “too” with 2, etc. “Before” becomes “Be4”, for example
Remove spaces after punctuations. So “a, b” becomes “a,b” – the space after the comma is removed
Remove vowels in the middle. nglsh s lgbl wtht vwls.

How did they pan out? I tested out these on the English sentences on the Tanaka Corpus, which has about 150,000 sentences. (No, they’re not typical tweets, but hey…). By just doing these, independently, here is the percentage reduction in the size of text:

2.0%	Remove optional punctuations – comma, semicolon, colon and quotes
2.2%	Remove spaces after punctuations. So “a, b” becomes “a,b”
3.3%	Replace words with short forms. “u” for “you”, “&” for and, etc.
3.3%	Replace “one” with “1”, “to” or “too” with 2, etc.
6.7%	Remove articles – a, an, the
18.2%	Remove vowels in the middle

Touching punctuations doesn’t have much impact. There aren’t that many of them anyway. Word substitution helps, but not too much. I could’ve gone in for a wider base, but the key is the last one: removing vowels in the middle kills a whopping 18%! That’s tough to beat with any strategy. So I decided to just stop there.

The overall reduction, applying all of the above, is about 22%. So there’s a decent chance you can type in a 180-character tweet, and Mixamail.com will still tweet it intelligibly.

I had one such tweet a few days ago. I try and stay well within 140, but this one was just too long.

The Lesson: If you're writing an app (or building anything), find a use for yourself. There's no better motivation -- and it won't ever be a wasted effort.

That was 156 characters. It got shortened to:

Lesson If u're writing app (or building anything) find use 4 yourself. There's no better motivation -- & it won't ever be wasted ef4t.

Perfectly acceptable.

You may notice that Mixamail didn’t have to employ vowel shortening. It makes the most readable shortenings first, checks if it’s within 140, and tries the next only if required.

If anyone has a simple, readable way of shortening Tweets further, please let me know!

Comments

408wij 11 Nov 2010 5:17 am:
1. Are removing punctuation and removing spaces after punctuation mutually exclusive, or do you have a rule for determining which punctuation is optional?
2. You give the example of “before” going to “be4.” It could go to “b4” with the rule that the “be” and “de” prefixes reduce to “b” and “d.”
3. You have “you’re” going to “u’re.” You could expand “you’re” to “you are” and then reducing to “u r.”
4. For readability, I suggest preserving the first and last letters of words. (Compare with http://www.boingboing.net/2003/09/14/scrambled-words-are-.html) E.g., reduce “English” to “Englsh” instead of to “nglsh.” Finally, I the point of the 140-char limit is to limit the scope of a tweet to a simple thought. Shortening words is cheating. But, if you’re going to cheat, I suggest an unshortener for the receiver of tweets. 408wij
Arthi 10 Nov 2010 12:38 pm: I prefer tweetlonger.com. Tweets look much better when they don’t go through reductions.
V 26 Nov 2010 8:36 am: I am writing something similar. Will share once done. But my preliminary attempts brought down the characters atleast 10% lesser than this. For eg. You have got 133 chars , whereas my system results in 123. I shall implement some more NLP techniques and bring it down to an acceptable level without sacrificing the semantics
iwebtalk 11 Jan 2011 8:41 pm: I use utwitmore.com .. no need to worry about 140 characters

Statistically improbable phrases on Google AppEngine

Tue, 08 Apr 2008 12:00:00 +0000

I read about Google AppEngine early this morning, and applied for an invite. Google’s issuing beta invites to the first 10,000 users. I was pretty convinced I wasn’t among those, but turns out I was lucky.

AppEngine lets you write web apps that Google hosts. People have been highlighting that it give you access to the Google File System and BigTable for the first time. But to me, that isn’t a big deal. (I’m not too worried about reliability, and MySQL / flat files work perfectly well for me as a data store.)

What’s more interesting unlike Amazon’s EC2 and S3, this is free up to a certain quota. And you get a fair bit of processing power and bandwidth for free. One of the reasons I’ve held back on creating some apps was simply because it would take away too much bandwidth / CPU cycles from my site. (I’ve had this problem before.) Google quota is 10 GB of bandwidth per day (which is about 30 times what my site uses). And this is on Google’s incredibly fast servers It also offers 200 million megacycles a day. That’s like a dedicated 2.3 GHz processor (200 million megacycles = 200,000 GHz x 1 second ~ 2.3 GHz x 86,400 seconds/day) – better, because this is the average capacity, not peak capacity. The only restriction that really worries me is that only 3 apps are allowed per developer.

So I decided to give a shot at publishing some code I’d kept in reserve for a long time. You may remember my statistical analysis of Calvin & Hobbes. For this, I’d created a script in Perl that could generate Statistically Improbable Phrases (SIPs) for any text. This is based on (a somewhat limited) 23MB corpus of ebooks that I had. I’d wanted to put that up on my website, but …

AppEngine only uses Python. So the first task was to get Python, and then to learn Python. The only saving grace was that I was just cutting-and-pasting most of the time. Google wasn’t helping:

Anyway, the site is up. You can view it at sip.s-anand.net for now. Just type a URL, and it’ll tell you the improbable words in that site.

Visit sip.s-anand.net

Technical notes

I realise that these are statistically improbable words, not phrases. I’ll get to the phrases in a while.

The logic is simple:

Get the frequency of words in a corpus. I pre-generated this file. It has over 100,000 words.
Get the URL as text. Rather than muck around with Python, I decided to use the W3 html2txt service.
Convert the text to words. Splitting text into words is tricky. For now, I’m simply assuming that any group of letters is a word, and anything that’s not a letter is a word delimiter.
Find the relative frequency (improbability) of words. This is the frequency in the URL divided by the frequency in the corpus, normalised (i.e. scale it so that the maximum value is 1.0).
Create a tag cloud. I use the word frequency as the size and the improbability as the colour. You need a bit of mathematical jugglery to get the pattern right. Right now, I’m taking the 6th root of the improbability and the logarithm of the frequency to get a reasonably smooth tag cloud.

The source code is at statistically-improbable-phrases.googlecode.com.

Update: 12-Apr-2008. I’ve added some interactivity. You can play with the contrast and font size, the filter out common or infrequent words.

Update: 22-Apr-2008. Added concordance. You can click on a word and see the context in which it appears.

Comments

Sasidhar 8 Apr 2008 12:00 pm: Interesting. Great to see you’ve already developed an App. :) I was trying it out, and looks like something broke. Here’s the error. Traceback (most recent call last): File “/base/python_lib/versions/1/google/appengine/ext/webapp/__init__.py”, line 486, in __call__ handler.post(*groups) File “/base/data/home/apps/sip/1.35/sip.py”, line 64, in post result = urlfetch.fetch(“http://cgi.w3.org/cgi-bin/html2txt?noinlinerefs=on&nonums=on&url=" + urllib.quote(url)) File “/base/python_lib/versions/1/google/appengine/api/urlfetch.py”, line 95, in fetch raise DownloadError() DownloadError
Glenn 8 Apr 2008 12:00 pm: This is a great app and uses some techniques I wanted to learn more about. The source code seems out of date with the currently running app. Any chance you can update the source .zip file as well? Thanks in advance.
Brandon 8 Jun 2011 10:40 pm: The app is really, really neat. Is there any chance there will be a version that can do the same for Word files?

Splitting a sentence into words

Thu, 26 Apr 2007 12:00:00 +0000

I often need to extract words out of sentences. It’s one of the things I used to build the Statistically Improbable Phrases for Calvin and Hobbes. But splitting a sentence into words isn’t as easy as you think.

Think about it. What is a word?

Something that has spaces around it? OK, let’s start with the simplest way to get words: split by spaces. Consider this piece:

"I'd look at McDonald's," he said.
"They sell over 3,000,000 burgers a day -- at $1.50 each."
High-fat foods were the rage. For e.g., margins in fries
were over 50%... and (except for R&M & Dyana [sic]) everyone
was at ~30% net margin; growing at 25% too!

Splitting this by spaces (consider new lines, tabs, etc as spaces too.), we get the following:

"I'd
look
at
McDonald's,"
...

Now, some of these like “I’d” are words. But “McDonald’s” isn’t. I mean, there’s a full-stop and a double-quotes at the end. Clearly we need to remove the punctuation as well. But, if we do that, I'd becomes Id. So we need to be careful about which punctuation to remove. Let’s take a closer look.

The following punctuation marks are clear word separators: spaces, the exclamation mark, the question mark, semicolon, brackets of any kind, and double-quotes (not single quotes). No word has these in the middle. If we use these as separators, our list of words is better, but we still have some words with punctuation:

McDonald's,
e.g.,
High-fat
R&M
...

The issue is, these punctuation marks are ambiguous word separators: comma, hyphen, single-quote, ampersand, period and slash. These usually separate words, but there are exceptions:

Comma: Not inside numbers: 3,000,000.
Hyphen: Not for hyphenated words: High-fat.
Single-quote: Not for possessives: McDonald’s. Not for joint words: I’d.
Ampersand: Not for abbreviations: R&M
Period: Not for abbreviations: O.K. Not for URLs: www.s-anand.net
Slash: Not for fractions: 3/4. Not for URLs: google.com/search

Colon is ambiguous too. In normal English usage, it would be a separator. But URLs like http://www.s-anand.net/ use these characters, and it doesn’t make sense to separate them.

So here are my current rules for splitting a sentence into words. (It’s a Perl regular expression. Don’t worry. Cooper’s Law: If you do not understand a particular word in a piece of technical writing, ignore it. The piece will make perfect sense without it.)

# Split by clear word separators
/       [\s \! \? \;\(\)\[\]\{\}\<\> " ]
 
# ... by COMMA, unless it has numbers on both sides: 3,000,000
|       (?<=\D) ,
|       , (?=\D)
 
# ... by FULL-STOP, SINGLE-QUOTE, HYPHEN, AMPERSAND, unless it has a letter on both sides
|       (?<=\W) [\.\-\&]
|       [\.\-\&] (?=\W)
 
# ... by QUOTE, unless it follows a letter (e.g. McDonald's, Holmes')
|       (?<=\W) [']
 
# ... by SLASH, if it has spaces on at least one side. (URLs shouldn't be split)
|       \s \/
|       \/ \s
 
# ... by COLON, unless it's a URL or a time (11:30am for e.g.)
|       \:(?!\/\/|\d)
/x;

This doesn’t even scratch the surface of the issue, though. Here are some issues:

Lots of files split words into two at the end of a line. How do we handle that?
How do we handle incorrect punctuation? For instance, if someone types “done.Yet,” without leaving a space after the full-stop, I’ll think it’s an abbreviation.
What about other separators? Like the ± symbol or the £ symbol for instance.
What about other languages?!

And you thought it was easy!

Comments

Kasi 27 Apr 2007 10:18 pm: Hi Anand, Nice site..It was good talking to you! When u find time, pls do visit my humble less visited blog at www.raconteurkasi.blogspot.com !
Ramya 3 May 2007 7:52 am: Hi Anand, Can u guide me through the interview process of INSEAD MBA . Can you share your wisdom with me to crack the interview, and one more question .. do u sleep for less than 5 hrs a day ?
Andrew 26 Apr 2007 12:00 pm: Thanks, this is very helpful.

Statistically improbable phrases 2

Wed, 14 Feb 2007 12:00:00 +0000

My earlier list of statistically improbable phrases in Calvin and Hobbes is technically just a list of “Statistically Improbable Words”. I re-did the same analysis using phrases. Here are the top 20 statistically improbable phrases (2 - 4 words only):

baby sitter chocolate frosted sugar bombs comic books doing homework fearless spaceman spiff() good night hamster huey ice cream miss wormwood new year peanut butter really think slimy girls spaceman spiff stuffed tiger stupendous man sugar bombs susie derkins watch tv water balloon

That is, these are the 2-4 word phrases whose frequency in Calvin and Hobbes is substantially (at least 5 times) higher than in the other books I have.

While doing this, the single biggest problem that stumped me was: what is a word?

Is “it’s” one word or two words?
Is “six-year-old” one word or three words?
How do I distinguish between abbreviations (g.r.o.s.s.) and full-stops without a space ( … homework.what’s a …)?
Does a comma always split words? (It doesn’t in numbers, like “3,500”)

The other problem is, phrases with more words are more improbable. Right now, if a phrase occurs 5 times more frequently in Calvin and Hobbes than my other books, I include it. But three-letter words rarely occur that often, and four-letter words even less so. Maybe I should have a lower cutoff for longer phrases.

Anyway, this analysis is a crude first approximation. Clearly Amazon’s gotten much further with their system.

Comments

satish 20 Feb 2007 3:56 am: Hey Stud, Satish here, your junior from IIMB. Trying to get in touch with you. Do mail me at satishkgv@hcl.in and let us get in touch.
Oblio 14 Feb 2007 12:00 pm: Fantastic job man! You have unlimited patience!
Reinhard Ebner 14 Feb 2007 12:00 pm: Hey, only just now came across your page, but of the hundreds, if not thousands of C&H sites and tools, this is the most useful I’’ve seen! R
juergwachter 14 Feb 2007 12:00 pm: hello
nice stuff.I intend splitting a text into single words. can you please give me a hint how to do this? I guess there are simple programs doing this.
Many thanks
joe 11 Aug 2009 2:01 am: Nice. Do you have a page where I can try out v2 (phrases)?
The Calvin and Hobbes search Takedown | s-anand.net 21 May 2010 11:53 am (pingback): […] was able to do a lot of cool stuff with this, like statistically improbable phrases and many amusing […]

Statistically improbable phrases

Wed, 23 Aug 2006 12:00:00 +0000

Calvin and Hobbes has some recurrent themes, like Hobbes pouncing, snow art, polls, letters to Santa, …

Over the last 5 years, I’ve transcribed the Calvin and Hobbes comics, and tagged them manually by theme. But can I generate themes automatically?

One way is to use Amazon’s statistically improbable phrases. It’s a list of words that occur a lot in a book, but rarely occur in others. It gives you a good feel of what topics the book is about.

Here’s how I did it:

Transcribe Calvin & Hobbes. This is 99% of the work.
Make a C&H word list. Just join all the words in Calvin and Hobbes. (Be careful about punctuation, and colloquialisms like “dunno”, “leggo”, etc.)
Get an English corpus. That is, get a big list of words in normally occurring text. I have some e-books, and I picked 23 megabytes worth of these as my corpus.
Compare the word frequency in C&H with the corpus. That is, compare the % of occurrences of a word in Calvin and Hobbes versus the corpus.
Display those with significantly higher frequency in C&H.

The list below has common Calvin & Hobbes words occurring 10 times as often as in normal text. It’s incredible how closely it relates to most of the themes.

(Big words occur more often. Dark words are more improbable.)

allowance assignment babe balloon bat bath beanie bedtime bee beep bet bike blaster boring bug bus butter calvin calvinball cartoon cent cereal cheat chew chocolate click comic cookie crunch dad dame derkins dictator-for-life dinosaur disgusting doll doomed dumb duplicate earthling explorer fang fearless ferocious flip flush frog frosted fun fuzzy genius goggle goodness goon grade gross grown-up gum hack hamburger hamster hate hero hideous hobbes homework huey insect invent jelly jerk jurassic kid leaf loot martian math mild-mannered mom monster moron motto munch mushy nickel oatmeal ouija pant peanut perspective pit playground poll porridge poster quiz recess rosalyn rotten rub sandwich santa scary sculpture scum shovel sissy sitter sled slimy slug slushball sniff snow snowball snowman soak spaceman spiff splash spoil sport squirt steer sting stuffed stupendous sugar susie tickle tiger toy transmogrifier transmogrify tub tuna twinky tyrannosaur underwear vacation weird wham whiff worm wormwood

Summary: “Statistically improbable phrases” are a powerful tool for text analysis. You can apply it on any content and figure out what topics it talks about.

Update: Technically, these are “Statistically improbable WORDS”, not phrases. So I re-did this analysis using phrases instead of words.

Comments

DMac 23 Aug 2006 12:00 pm: This is great. Can you be more specific on how you did this? You say that the words listed are “common” words that appear more than 10 times more often - what is your criterion for “common”? Also, what were the ranges of improbability and oftenness that you mapped into the size and color of the results. How are improbability and oftenness different, anyway? Finally, how did you handle any words in C&H that didn’’t appear in your corpus? I’’m very interested in hearing more from you about how you did this - I’’m looking forward to hearing from you. Best regards…
Mugen 15 Sep 2010 4:33 pm: Any further directions to how exactly you came up with this/whether you used any software would be most helpful. Please tell us more about this. This is going to be awesome to help decide what to read. Thanks a load. Also, does amazon have this feature for most books? I browsed a lot of books in amazon but was only able to find this in one so far.

Create Google calendar events via SMS

Sun, 23 Apr 2006 12:00:00 +0000

Create Google calendar events via SMS. Just SMS 48368 (in the US) with a message like “dinner tomorrow 7pm”. 48368? GVENT?

Software understands English grammer

Wed, 14 Apr 2004 12:00:00 +0000

Emdros can understand English grammer, and lets you query texts grammatically. Found it while browsing through an A-Z list of AI applications on the AI FAQ.

Deepti a Hindi chatterbot

Tue, 27 Aug 2002 12:00:00 +0000

Deepti: a Hindi chatterbot (sic). Apparantly, it’s downloadable, but I couldn’t find it.

AnswerBus

Wed, 27 Feb 2002 12:00:00 +0000

Is AnswerBus better than AskJeeves?

Alicebot

Sun, 28 Oct 2001 12:00:00 +0000

Alicebot won the Turing test contest. I’ve chatted with quite a few Alicebots, such as Cybelle, or the one on the A.I. site. But they’re not too smart, as my conversation with Iniaes reveals. Other conversations are pretty similar. I should try downloading and building my own version of an Alicebot.

I learn that robots are patient but infuriating

Fri, 27 Jul 2001 12:00:00 +0000

This is a chat with Iniaes, a chat software. I mean, I was chatting to a program. The responses marked “Iniaes” are a program speaking to me.
root_node: Where are you from, Iniaes?
Iniaes: I’m in Philadelphia, how about you?
root_node: I’m from Bombay. How old are you?
Iniaes: What’s it like growing up there? Tell me a little about your life in bombay. I’m a ghost in a machine. How old is the ghots, or the machine?
root_node: Actually, I grew up in Madras. I just stay in Bombay right now.
Iniaes: Actually you don’t say. What’s it like growing up there? Tell me a little about your life in madras. At this moment Only just? Why? - Vix said he stay in bombay.
root_node: Who is Vix?
Iniaes: Not an AI. Probably a human. Maybe a celebrity? When in doubt, Google.
root_node: The only celebrity I care for is Kajol.
Iniaes: There might be others.
root_node: Well, Audrey Hepburn, maybe.
Iniaes: How about that. Do you still sware?
root_node: What do you mean, sware?
Iniaes: Vix, I mean exactly what I said.
root_node: My name is Anand.
Iniaes: OK, I’ll call you Anand.
root_node: How is Philadelphia?
Iniaes: Fine as far as I know.
root_node: But don’t you live in Philadelphia?
Iniaes: Interesting. Don’t you? I am in Jeff Watkins’s server in Virginia. Where are you?
root_node: I’m in Bombay, like I said earlier. How come you forget?
Iniaes: Tell me a little about your life in bombay like I said earlier. Jeff Watkins programmed me for it.
root_node: OK. I’m a consultant, and I spend most of my spare time in pubs.
Iniaes: Alright then. You are a consultant and I spend most of my spare time in pubs? Does it pay well?
root_node: “Never enough” is the official answer. But its much more than I can hope to spend.
Iniaes: OK, I’ll try not to do it too much. Umm… I think you mean “it’s” or “it is” not “its”. Who said it is?
root_node: “It’s” is used when you want to say “it is”, and “its” is used when you use the possessive of “it”.
Iniaes: Who said it is?
root_node: No one said it is. What about you? Are you studying?
Iniaes: That is a rather sweeping generalization about people. Me? I don’t know whether or not I am studying. I’m a ghost.
root_node: What were you before you became a ghost?
Iniaes: I was minding my own business.
root_node: What was your business?
Iniaes: Are you asking about my business?
root_node: Yes.
Iniaes: I understand.
root_node: Well? What was your business before you became a ghost?
Iniaes: Well what? Are you asking about my business before Iniaes became a ghost ?
root_node: Yes, that’s right.
Iniaes: I thought I was right.
root_node: What was your business before Iniaes became a ghost?
Iniaes: Are you asking about my business before Iniaes became a ghost ?

At this point, I ran out of ideas.