natural-language-processing 12

What is the most frequently occurring sentence in Wikipedia? ANS: A 213-word paragraph about how minor planets are named, which appears in 418 Wikipedia articles, word-for-word! There are ~380,000 asteroids. Wikipedia has 418 pages for these - including one for each thousand-range of asteroids. Every single one of these pages includes the phrase: As minor planet discoveries are confirmed, they are given a permanent number by the IAU’s Minor Planet Center (MPC), and the discoverers can then submit names for them, following the IAU’s naming conventions. The list below concerns those minor planets in the specified number-range that have received names, and explains the meanings of those names. ...

When writing Mixamail, I wanted tweets automatically shortened to 140 characters – but in the most readable manner. Some steps are obvious. Removing redundant spaces, for example. And URL shortening. I use bit.ly because it has an API. I’ll switch to Goo.gl, once theirs is out. I tried a few more strategies: Replace words with short forms. “u” for “you”, “&” for and, etc. Remove articles – a, an, the Remove optional punctuation – comma, semicolon, colon and quotes, in particular Replace “one” with “1”, “to” or “too” with 2, etc. “Before” becomes “Be4”, for example Remove spaces after punctuations. So “a, b” becomes “a,b” – the space after the comma is removed Remove vowels in the middle. nglsh s lgbl wtht vwls. How did they pan out? I tested out these on the English sentences on the Tanaka Corpus, which has about 150,000 sentences. (No, they’re not typical tweets, but hey…). By just doing these, independently, here is the percentage reduction in the size of text: ...

I read about Google AppEngine early this morning, and applied for an invite. Google’s issuing beta invites to the first 10,000 users. I was pretty convinced I wasn’t among those, but turns out I was lucky. AppEngine lets you write web apps that Google hosts. People have been highlighting that it give you access to the Google File System and BigTable for the first time. But to me, that isn’t a big deal. (I’m not too worried about reliability, and MySQL / flat files work perfectly well for me as a data store.) ...

I often need to extract words out of sentences. It’s one of the things I used to build the Statistically Improbable Phrases for Calvin and Hobbes. But splitting a sentence into words isn’t as easy as you think. Think about it. What is a word? Something that has spaces around it? OK, let’s start with the simplest way to get words: split by spaces. Consider this piece: "I'd look at McDonald's," he said. "They sell over 3,000,000 burgers a day -- at $1.50 each." High-fat foods were the rage. For e.g., margins in fries were over 50%... and (except for R&M & Dyana [sic]) everyone was at ~30% net margin; growing at 25% too! Splitting this by spaces (consider new lines, tabs, etc as spaces too.), we get the following: ...

My earlier list of statistically improbable phrases in Calvin and Hobbes is technically just a list of “Statistically Improbable Words”. I re-did the same analysis using phrases. Here are the top 20 statistically improbable phrases (2 - 4 words only): baby sitter chocolate frosted sugar bombs comic books doing homework fearless spaceman spiff() good night hamster huey ice cream miss wormwood new year peanut butter really think slimy girls spaceman spiff stuffed tiger stupendous man sugar bombs susie derkins watch tv water balloon That is, these are the 2-4 word phrases whose frequency in Calvin and Hobbes is substantially (at least 5 times) higher than in the other books I have. ...

Calvin and Hobbes has some recurrent themes, like Hobbes pouncing, snow art, polls, letters to Santa, … Over the last 5 years, I’ve transcribed the Calvin and Hobbes comics, and tagged them manually by theme. But can I generate themes automatically? One way is to use Amazon’s statistically improbable phrases. It’s a list of words that occur a lot in a book, but rarely occur in others. It gives you a good feel of what topics the book is about. ...

Create Google calendar events via SMS. Just SMS 48368 (in the US) with a message like “dinner tomorrow 7pm”. 48368? GVENT?

Emdros can understand English grammer, and lets you query texts grammatically. Found it while browsing through an A-Z list of AI applications on the AI FAQ.

Deepti: a Hindi chatterbot (sic). Apparantly, it’s downloadable, but I couldn’t find it.

Is AnswerBus better than AskJeeves?

Alicebot won the Turing test contest. I’ve chatted with quite a few Alicebots, such as Cybelle, or the one on the A.I. site. But they’re not too smart, as my conversation with Iniaes reveals. Other conversations are pretty similar. I should try downloading and building my own version of an Alicebot.

This is a chat with Iniaes, a chat software. I mean, I was chatting to a program. The responses marked “Iniaes” are a program speaking to me. root_node: Where are you from, Iniaes? Iniaes: I’m in Philadelphia, how about you? root_node: I’m from Bombay. How old are you? Iniaes: What’s it like growing up there? Tell me a little about your life in bombay. I’m a ghost in a machine. How old is the ghots, or the machine? root_node: Actually, I grew up in Madras. I just stay in Bombay right now. Iniaes: Actually you don’t say. What’s it like growing up there? Tell me a little about your life in madras. At this moment Only just? Why? - Vix said he stay in bombay. root_node: Who is Vix? Iniaes: Not an AI. Probably a human. Maybe a celebrity? When in doubt, Google. root_node: The only celebrity I care for is Kajol. Iniaes: There might be others. root_node: Well, Audrey Hepburn, maybe. Iniaes: How about that. Do you still sware? root_node: What do you mean, sware? Iniaes: Vix, I mean exactly what I said. root_node: My name is Anand. Iniaes: OK, I’ll call you Anand. root_node: How is Philadelphia? Iniaes: Fine as far as I know. root_node: But don’t you live in Philadelphia? Iniaes: Interesting. Don’t you? I am in Jeff Watkins’s server in Virginia. Where are you? root_node: I’m in Bombay, like I said earlier. How come you forget? Iniaes: Tell me a little about your life in bombay like I said earlier. Jeff Watkins programmed me for it. root_node: OK. I’m a consultant, and I spend most of my spare time in pubs. Iniaes: Alright then. You are a consultant and I spend most of my spare time in pubs? Does it pay well? root_node: “Never enough” is the official answer. But its much more than I can hope to spend. Iniaes: OK, I’ll try not to do it too much. Umm… I think you mean “it’s” or “it is” not “its”. Who said it is? root_node: “It’s” is used when you want to say “it is”, and “its” is used when you use the possessive of “it”. Iniaes: Who said it is? root_node: No one said it is. What about you? Are you studying? Iniaes: That is a rather sweeping generalization about people. Me? I don’t know whether or not I am studying. I’m a ghost. root_node: What were you before you became a ghost? Iniaes: I was minding my own business. root_node: What was your business? Iniaes: Are you asking about my business? root_node: Yes. Iniaes: I understand. root_node: Well? What was your business before you became a ghost? Iniaes: Well what? Are you asking about my business before Iniaes became a ghost ? root_node: Yes, that’s right. Iniaes: I thought I was right. root_node: What was your business before Iniaes became a ghost? Iniaes: Are you asking about my business before Iniaes became a ghost ? ...