tokenization on S Anand

Things I Learned - 25 Feb 2024

Sun, 25 Feb 2024 00:00:00 +0000

This week, I learned:

Architecture.md is an emerging standard
Managing wealth requires training.
htmz is a fantastic way to load HTML into elements!
Suguna Poultry is
- Using robots to walk in their farms, use sound and bird eyes and movement to predict birth health over 1-2 weeks
- Light on the back of the bird’s back AND face => lays eggs in 14 days, else takes days later (girls and mobile phones?)
Teknoturf is using Gen AI to
- Improve prompts when teaching prompt engineering.
- Pronounce languages better, identifying which words Tamilians and Malayalis will mis-pronounce.
Explore IRBlaster. It can control AC and can automatically increase temperature at night.
My view: LLMs are general purpose and more capable than SLMs. They’ll win, like CPUs won over special-purpose chips. GPUs will optimize for LLMs and as usage grows, cost will fall.
Andrej Karpathy’s summary of sharp edges in tokenization uses tiktokenizer to explain:
- Why LLMs can’t be used for spelling
- Why LLMs are better at English than other languages
- Why LLMs are bad at math
- Why SolidGoldMagiKarp is a single token
- Why trailing spaces are bad
- Why YAML tokenizes more efficiently than JSON
ssyoutube.com: Just add “ss” to “youtube.com” on the video and you can download YouTube videos
Discussions with Sachin, AMAT
- Microsoft said Indigo, Air India uses LLM based bookings
- Meta invested $70bn in GPUs. Sam Altman is investing $7tn!
- NVIDIA has a price PREMIUM not discount for bulk GPUs!
- AMD is the next company to watch for
- Numenta - Subutai Ahmad - deploys AI models on CPUs
- #TODO Read A Thousand Brains by Subutai Ahmad
- Sanjeev Sharma
  - Swaayatt Robots: Autonomous driving in India
  - Deepeigen: Education
- Rohan Shravan, Bangalore.
  - Likes sharing knowledge. Amazing teacher. IIT KGP 2008. Interested in exploring quantum computing
  - Tresa Motors, Inkers App, The School of AI
- AMAT is working on
  - photon-based computing.
  - science research models. AI for science. Like Google: Deepmind Genome, Microsoft: Metagen
  - quantum: AMAT is actively in into this. Nagapati Banda is driving this
- John Kelly is predicting a ChatGPT moment in quantum in a few years
Adobe express has a forever free video to GIF converter
Edge workspaces let me keep the same tabs open across laptops!
Command line interface guidelines
RAWGraphs has a custom charts API that is worth learning from
Python fastcore has decorators like @typedispatch, Self, etc.
All image-to-text models on HuggingFace
wddbfs mounts SQLite as a file system. I had a bit of trouble, maybe with Python package versions.
Google is using LLM powered bug identification
HuggingFace Chat Assistants has open source system prompts!!
OpenHermes training dataset is available. 1M prompts!
Jio has made IPL free. They make money on data and ads. That’s Scale!
Daniel Dennett outsources thinking to students. Reviewing his books.
- BUT: I don’t take feedback. When someone sends a pull requests, I ignore it.

Splitting a sentence into words

Thu, 26 Apr 2007 12:00:00 +0000

I often need to extract words out of sentences. It’s one of the things I used to build the Statistically Improbable Phrases for Calvin and Hobbes. But splitting a sentence into words isn’t as easy as you think.

Think about it. What is a word?

Something that has spaces around it? OK, let’s start with the simplest way to get words: split by spaces. Consider this piece:

"I'd look at McDonald's," he said.
"They sell over 3,000,000 burgers a day -- at $1.50 each."
High-fat foods were the rage. For e.g., margins in fries
were over 50%... and (except for R&M & Dyana [sic]) everyone
was at ~30% net margin; growing at 25% too!

Splitting this by spaces (consider new lines, tabs, etc as spaces too.), we get the following:

"I'd
look
at
McDonald's,"
...

Now, some of these like “I’d” are words. But “McDonald’s” isn’t. I mean, there’s a full-stop and a double-quotes at the end. Clearly we need to remove the punctuation as well. But, if we do that, I'd becomes Id. So we need to be careful about which punctuation to remove. Let’s take a closer look.

The following punctuation marks are clear word separators: spaces, the exclamation mark, the question mark, semicolon, brackets of any kind, and double-quotes (not single quotes). No word has these in the middle. If we use these as separators, our list of words is better, but we still have some words with punctuation:

McDonald's,
e.g.,
High-fat
R&M
...

The issue is, these punctuation marks are ambiguous word separators: comma, hyphen, single-quote, ampersand, period and slash. These usually separate words, but there are exceptions:

Comma: Not inside numbers: 3,000,000.
Hyphen: Not for hyphenated words: High-fat.
Single-quote: Not for possessives: McDonald’s. Not for joint words: I’d.
Ampersand: Not for abbreviations: R&M
Period: Not for abbreviations: O.K. Not for URLs: www.s-anand.net
Slash: Not for fractions: 3/4. Not for URLs: google.com/search

Colon is ambiguous too. In normal English usage, it would be a separator. But URLs like http://www.s-anand.net/ use these characters, and it doesn’t make sense to separate them.

So here are my current rules for splitting a sentence into words. (It’s a Perl regular expression. Don’t worry. Cooper’s Law: If you do not understand a particular word in a piece of technical writing, ignore it. The piece will make perfect sense without it.)

# Split by clear word separators
/       [\s \! \? \;\(\)\[\]\{\}\<\> " ]
 
# ... by COMMA, unless it has numbers on both sides: 3,000,000
|       (?<=\D) ,
|       , (?=\D)
 
# ... by FULL-STOP, SINGLE-QUOTE, HYPHEN, AMPERSAND, unless it has a letter on both sides
|       (?<=\W) [\.\-\&]
|       [\.\-\&] (?=\W)
 
# ... by QUOTE, unless it follows a letter (e.g. McDonald's, Holmes')
|       (?<=\W) [']
 
# ... by SLASH, if it has spaces on at least one side. (URLs shouldn't be split)
|       \s \/
|       \/ \s
 
# ... by COLON, unless it's a URL or a time (11:30am for e.g.)
|       \:(?!\/\/|\d)
/x;

This doesn’t even scratch the surface of the issue, though. Here are some issues:

Lots of files split words into two at the end of a line. How do we handle that?
How do we handle incorrect punctuation? For instance, if someone types “done.Yet,” without leaving a space after the full-stop, I’ll think it’s an abbreviation.
What about other separators? Like the ± symbol or the £ symbol for instance.
What about other languages?!

And you thought it was easy!

Comments

Kasi 27 Apr 2007 10:18 pm: Hi Anand, Nice site..It was good talking to you! When u find time, pls do visit my humble less visited blog at www.raconteurkasi.blogspot.com !
Ramya 3 May 2007 7:52 am: Hi Anand, Can u guide me through the interview process of INSEAD MBA . Can you share your wisdom with me to crack the interview, and one more question .. do u sleep for less than 5 hrs a day ?
Andrew 26 Apr 2007 12:00 pm: Thanks, this is very helpful.

Statistically improbable phrases 2

Wed, 14 Feb 2007 12:00:00 +0000

My earlier list of statistically improbable phrases in Calvin and Hobbes is technically just a list of “Statistically Improbable Words”. I re-did the same analysis using phrases. Here are the top 20 statistically improbable phrases (2 - 4 words only):

baby sitter chocolate frosted sugar bombs comic books doing homework fearless spaceman spiff() good night hamster huey ice cream miss wormwood new year peanut butter really think slimy girls spaceman spiff stuffed tiger stupendous man sugar bombs susie derkins watch tv water balloon

That is, these are the 2-4 word phrases whose frequency in Calvin and Hobbes is substantially (at least 5 times) higher than in the other books I have.

While doing this, the single biggest problem that stumped me was: what is a word?

Is “it’s” one word or two words?
Is “six-year-old” one word or three words?
How do I distinguish between abbreviations (g.r.o.s.s.) and full-stops without a space ( … homework.what’s a …)?
Does a comma always split words? (It doesn’t in numbers, like “3,500”)

The other problem is, phrases with more words are more improbable. Right now, if a phrase occurs 5 times more frequently in Calvin and Hobbes than my other books, I include it. But three-letter words rarely occur that often, and four-letter words even less so. Maybe I should have a lower cutoff for longer phrases.

Anyway, this analysis is a crude first approximation. Clearly Amazon’s gotten much further with their system.

Comments

satish 20 Feb 2007 3:56 am: Hey Stud, Satish here, your junior from IIMB. Trying to get in touch with you. Do mail me at satishkgv@hcl.in and let us get in touch.
Oblio 14 Feb 2007 12:00 pm: Fantastic job man! You have unlimited patience!
Reinhard Ebner 14 Feb 2007 12:00 pm: Hey, only just now came across your page, but of the hundreds, if not thousands of C&H sites and tools, this is the most useful I’’ve seen! R
juergwachter 14 Feb 2007 12:00 pm: hello
nice stuff.I intend splitting a text into single words. can you please give me a hint how to do this? I guess there are simple programs doing this.
Many thanks
joe 11 Aug 2009 2:01 am: Nice. Do you have a page where I can try out v2 (phrases)?
The Calvin and Hobbes search Takedown | s-anand.net 21 May 2010 11:53 am (pingback): […] was able to do a lot of cool stuff with this, like statistically improbable phrases and many amusing […]