tokenization 3

This week, I learned: Architecture.md is an emerging standard Managing wealth requires training. htmz is a fantastic way to load HTML into elements! Suguna Poultry is Using robots to walk in their farms, use sound and bird eyes and movement to predict birth health over 1-2 weeks Light on the back of the bird’s back AND face => lays eggs in 14 days, else takes days later (girls and mobile phones?) Teknoturf is using Gen AI to Improve prompts when teaching prompt engineering. Pronounce languages better, identifying which words Tamilians and Malayalis will mis-pronounce. Explore IRBlaster. It can control AC and can automatically increase temperature at night. My view: LLMs are general purpose and more capable than SLMs. They’ll win, like CPUs won over special-purpose chips. GPUs will optimize for LLMs and as usage grows, cost will fall. Andrej Karpathy’s summary of sharp edges in tokenization uses tiktokenizer to explain: Why LLMs can’t be used for spelling Why LLMs are better at English than other languages Why LLMs are bad at math Why SolidGoldMagiKarp is a single token Why trailing spaces are bad Why YAML tokenizes more efficiently than JSON ssyoutube.com: Just add “ss” to “youtube.com” on the video and you can download YouTube videos Discussions with Sachin, AMAT Microsoft said Indigo, Air India uses LLM based bookings Meta invested $70bn in GPUs. Sam Altman is investing $7tn! NVIDIA has a price PREMIUM not discount for bulk GPUs! AMD is the next company to watch for Numenta - Subutai Ahmad - deploys AI models on CPUs #TODO Read A Thousand Brains by Subutai Ahmad Sanjeev Sharma Swaayatt Robots: Autonomous driving in India Deepeigen: Education Rohan Shravan, Bangalore. Likes sharing knowledge. Amazing teacher. IIT KGP 2008. Interested in exploring quantum computing Tresa Motors, Inkers App, The School of AI AMAT is working on photon-based computing. science research models. AI for science. Like Google: Deepmind Genome, Microsoft: Metagen quantum: AMAT is actively in into this. Nagapati Banda is driving this John Kelly is predicting a ChatGPT moment in quantum in a few years Adobe express has a forever free video to GIF converter Edge workspaces let me keep the same tabs open across laptops! Command line interface guidelines RAWGraphs has a custom charts API that is worth learning from Python fastcore has decorators like @typedispatch, Self, etc. All image-to-text models on HuggingFace wddbfs mounts SQLite as a file system. I had a bit of trouble, maybe with Python package versions. Google is using LLM powered bug identification HuggingFace Chat Assistants has open source system prompts!! OpenHermes training dataset is available. 1M prompts! Jio has made IPL free. They make money on data and ads. That’s Scale! Daniel Dennett outsources thinking to students. Reviewing his books. BUT: I don’t take feedback. When someone sends a pull requests, I ignore it.

I often need to extract words out of sentences. It’s one of the things I used to build the Statistically Improbable Phrases for Calvin and Hobbes. But splitting a sentence into words isn’t as easy as you think. Think about it. What is a word? Something that has spaces around it? OK, let’s start with the simplest way to get words: split by spaces. Consider this piece: "I'd look at McDonald's," he said. "They sell over 3,000,000 burgers a day -- at $1.50 each." High-fat foods were the rage. For e.g., margins in fries were over 50%... and (except for R&M & Dyana [sic]) everyone was at ~30% net margin; growing at 25% too! Splitting this by spaces (consider new lines, tabs, etc as spaces too.), we get the following: ...

My earlier list of statistically improbable phrases in Calvin and Hobbes is technically just a list of “Statistically Improbable Words”. I re-did the same analysis using phrases. Here are the top 20 statistically improbable phrases (2 - 4 words only): baby sitter chocolate frosted sugar bombs comic books doing homework fearless spaceman spiff() good night hamster huey ice cream miss wormwood new year peanut butter really think slimy girls spaceman spiff stuffed tiger stupendous man sugar bombs susie derkins watch tv water balloon That is, these are the 2-4 word phrases whose frequency in Calvin and Hobbes is substantially (at least 5 times) higher than in the other books I have. ...