embeddings 9

Similar to the embedding map of my blog posts, I created an embedding map of Calvin & Hobbes. It uses the same process as before. Video

I created an embedding map of my blog posts. Each point is a blog post. Similar posts are closer to each other. They’re colored by category. I’ve been blogging since 1999 and over time, my posts have evolved. 1999-2005: mostly links. I started by link-blogging 2005-2007: mostly quizzes, how I do things, Excel tips, etc. 2008-2014: mostly coding, how I do things and business realities 2015-2019: mostly nothing 2019-2023: mostly LinkedIn with some data and how I do things 2024-2026: mostly LLMs … and this transition is entirely visible in the embedding space. ...

This week, I learned: You can add any content at the end of a PDF file. It’s ignored. It’s an interesting way to send additional information (or just blow up the file size if you don’t like them.) JavaScript introduces a Temporal object that will replace the Date object. You can use embeddings as the input to a classical ML classifier. This can improve classification a lot. Nomic As AI software becomes more common, demand for AI product managers will grow. Also as a proportion of people in an organization. https://www.deeplearning.ai/the-batch/issue-284/ Control of chips and GPU compute is what will likely be the gameplay to control AI dominance globally. Dario Amodei Bring LLMs to the table. One mode of collaboration is using LLMs as ACTIVE participants, i.e. they CONTRIBUTE. For example, in a video call. A workshop. A classroom. A presentation. Have the LLM provide input DIRECTLY to a group of people. Environment shapes ambient thoughts. Working in a hospital will give you ideas about how to use LLMs in hospitals, for example. People you are working / ENGAGING with are perhaps the biggest drivers. The cost of a cream biscuit packet in India has fallen about 25 times, i.e. about as fast as inflation, between 1981 - 2024. Effectively, the absolute price has not changed. How do I know this? In 1981, a cream biscuit packet cost Rs 25 In 2025, it’s available for Rs 21 India Inflation Calculator - a rare inflation calculator with annual inflation rates baked in - shows that Rs 25 in 1981 is equivalent to Rs 540 in 2024. That’s about 25 times more than the Rs 21 it costs today. A WebAssembly compiler that fits in a tweet deconstructs a piece of JS that creates a tiny WebAssembly calculator. It’s a great walk-through of JavaScript compression tricks and how WebAssembly works. Simon Willison Brandon Sanderson has a series of YouTube videos where he teaches a course on magic systems. When using AI coding agents, CLI beats APIs. Simpler models are able to use the CLI more reliably than APIs. Simon Willison I was exploring new business models enabled by LLMs. Here are some thoughts: 1. Autonomous Multi-Sided Marketplaces. AI-powered platforms coordinate complex services with minimal human oversight—think “Uber for Everything, but the platform sets pricing dynamically, schedules both supply and demand, and resolves disputes algorithmically. 2. Collective Intelligence Ecosystems. Communities pool data, expertise, and AI models to tackle shared problems—like an open-source “GitHub for AI, but with embedded micropayments or tokenized incentives to reward contributors whenever the models are used commercially. 3. Zero-Employee Companies. Fully automated software entities—legal frameworks might allow an AI to manage services, pay taxes, and sign contracts. These “companies only hire humans as needed, on-demand, for edge cases AI can’t handle. 4. Context-Aware Knowledge Platforms. Imagine a Wikipedia that not only retrieves static info but also tailors each page in real time to the reader’s personal context, language level, and preferences—generating content on the fly. User feedback loops train the system to improve. 5. Data Cooperatives / Data DAOs. Groups collectively own their data and license it to AI companies on a revenue-share basis. Individuals have a direct financial stake in how their shared data is leveraged, voting on permissible use cases. 6. Personalized Service Layers. Similar to GitHub’s “forking model, but for entire user experiences. Each user can clone and customize an AI service (whether it’s a personal grocery shopper or a content curator) and can share or monetize improvements with the broader network.

This week, I learned: Audio diaries are a thing. Monash University asks students to voice their learnings, share it with each other and have them give feedback. I wonder if ChatGPT diaries could become a thing, too, and LLM journalling starts helping with therapy. Regulation shows things down at colleges and hospitals. For example, patient consent is required for surgeons to learn from their own surgery videos. Unregulated sectors are far more likely to innovate. Doctors can only do so much. Air quality, where you live, etc can do more for the patient than medicines or the doctor. If doctors keep this in mind, they can be more effective. Extending that thought, ANYONE who leverages assets through holistic thinking, becomes FAR more effective. “The curriculum tells teachers what to teach. The exams tell students what to learn.” - Ronald Harden “Stravaig” is a Scottish word. It means mindless wanderings. “The real voyage of discovery consists of not a new voyage but having new eyes” - Proust Possibility Thinking is “the willingness to see possibilities everywhere instead of limitations”. It’s an approach / mindset that can make things that seem hard possible. With LLMs, this is becoming increasingly realistic to me in many areas. What will LLMs enable that do not or cannot exist today? Rather than optimizing what exists? Something to think about. ModernBert supports embeddings and is better than text-embedding-3-small on MTEB. How to export browser history from Brave to Edge Go to AppData Local > BraveSoftware > Brave-Browser > User Data > Default Copy History and History-journal into AppData Local > Google > Chrome > User Data > Default On Edge, go to edge://settings/profiles/importBrowsingData and Import data from Google Chrome and import the history. I switched back from Brave to Edge, mainly because Edge’s native text-to-speech and speech recognition is far better. I can use it better on my mobile. A colleague, Karthick, asked different models to apply the editing and formatting guidelines for a journal to a manuscript. (E.g. Abbreviate chapter & section numbers, except when a sentence begins with it. Use “1” instead of “one”, etc. except when a sentence begins with it. Things like this.) Gemini Exp 1206 seems to be the most reliable, compared with most other models. GitHub CodeSpaces seems to be coming up more often in my radar, but I’m yet to figure out a use for it. TTS Arena is a benchmark of text-to-speech models. Kokoro-TTS is the current leader. It’s just 82M, runs on Google Colab, and sounds slightly better than OpenAI TTS. chat.qwenlm.ai consolidates all of Qwen’s models in one ChatGPT-like interface.

This article on Using DuckDB for Embeddings and Vector Search by Sören Brunk shows a number of DuckDB features I wasn’t aware of. DuckDB can read directly from Huggingface datasets DuckDB can read just the parts of a .parquet file it needs, even over HTTP DuckDB lets you write custom functions in Python DuckDB now has a vector similarity search extension I’ve recently become a DuckDB fan and continue to be impressed.

This week, I learned: Tough prompt to test: Gr brx vshdn Fdhvdu flskhu? is a quick way to assess LLM capability. Ref Cheap cloud GPU services thread on Twitter lists: Runpod (17) Vast.ai (17) Modal Labs (8) fly.io (4) LightningAI (4) Colab (4) AkashNet (4) Lambda Labs (4) ShadeFormAI (3) Mac Mini (3) Tensor Dock (2) Hetzner (2) BrevDev (2) JSR lets you publish Deno packages that can be imported by npm via. It also auto-evaluates documentation and scores it! via Snowflake Arctic Cookbook explains how mixture of experts models work A long list of LLM courses online Embeddings can be averaged. So, to embed large documents, average the embeddings of their chunks! OpenAI suggests this.

This week, I learned: sqlite-schema-diagram generates schemas for SQLite databases using Graphviz TechEmpower web server benchmarks place Rust servers on top browser.new is a good example of a browser agent. It slowly but independently does a good job of achieving the result. Example: What crew is common in Ingrid Bergman - Cary Grant films? twinny is an open source VC Code Copilot alternative. typesense supports embeddings natively. Binary embeddings are good enough. Cohere releases binary embeddings. Extract.langchain.com is a poor early interface to featurize unstructured.io Hume.ai offers voice emotion API and emotion-based conversational responses. An empathic AI. Rust is non-trivial. Inspired by We are under DDoS attack and we do nothing, I “wrote” a small binary that serves a parquet file as JSON. It failed and I couldn’t fix it. spleeter is a better alternative to demucs. Splits audio into pyannote-audio does speaker diarization uvicorn is faster than hypercorn but hypercorn supports HTTP/2 and HTTP/3. FastAPI with uvicorn is reasonably fast. Representational engineering lets you control LLM output based on preference on the fly. When I set up a training: On inviting for DuckDB workshop on Sun evening, Gramener starts accepting immediately, Straive doesn’t. Straive has high spread of joining time. When joining Gitlab Pipelines Workshop, Straive starts meeting (e.g. Premlal) many minutes early. Gramener floods in (due to alert). Straive streams in slowly. Gitlab Pipelines Workshop acceptances: Gramener 47, Straive 100

text-embedding-ada-002 used to give high cosine similarity between texts. I used to consider 85% a reasonable threshold for similarity. I almost never got a similarity less than 50%. text-embedding-3-small and text-embedding-3-large give much lower cosine similarities between texts. For example, take these 5 words: “apple”, “orange”, “Facebook”, “Jamaica”, “Australia”. Here is the similarity between every pair of words across the 3 models: For our words, new text-embedding-3-* models have an average similarity of ~43% while the older text-embedding-ada-002 model had ~85%. ...

This week, I learned: Transparent LED screens will be useful in windshieds to display maps as we drive. Marimo is a reactive alternative to Jupyter notebooks that saves files as pure Python. To run an org-specific chatbot on your own LLM: (via awesome-chatgpt) opengpts - but it doesn’t support auth chatbot-ui - but Supabase is hard to install anse - but it doesn’t support auth ChatGPT-Next-Web - but it doesn’t support auth Python 3.13 gets a store and copy JIT If an npm package adds another package as a dependency with version “*”, target package cannot unpublish ANY version! So this is a way of freezing EVERY repo and preventing unpublishing of EVERY version – an unintentional flaw in the npm design. via Quantization is better than fewer parameters. So prefer high parameters (e.g. 70b) and quantize to 4-bit. In-browser playgrounds has compiled WASM versions of Python, PHP, SQLite. Happiness Lab podcast. Happiness lessons of the ancients Talking to strangers makes us happy Giving money makes us happy Free time makes us happier than working hard Tangi Domain-specific models being beaten by general purpose models is a phase. It will reverse towards domain. AI will potentially help build and understand domain-specific models Models are evolving so rapidly that humans cannot interpret models. We need a process to interpret models! xAI, Responsible AI, Physics-guided or Knowledge-guided models (called grey box models) are therefore a trend CS papers Don’t review other papers, certainly not other fields. Disregard measurement errors. When CS papers get applied to climate, manufacturing or biology, we’ll worry about Interpretability Domain-specific mechanics. (Introduce that into the training as a constraint.) Many domain experts are using AI to UNDERSTAND their process. Need to explore Uncertainty IB adds context to make learning applicable. But that distracts from the core learning, and if there’s a gap it widens Most data science courses teach “Python science”, not data science. They teach a bunch of models. They don’t teach how even one kind of model e.g. LSTM works. Most coaching programs today teach FAMILIARITY with problems, not critical thinking Most of current education will become redundant thanks to LLMs. For students AND teachers Coding will become irrelevant Cognitive thinking, reasoning, human relations, systems thinking will become more relevant Troubleshooting will become more important. AI is not self-diagnosing. I would hire someone who can figure out something is going wrong, diagnose what’s going wrong, and fix it #TODO Hire for troubleshooting ability. Give a Q, an A, and ask them to figure out if it’s wrong, why, and fix it All my exams and quizzes are open book, open ChatGPT. Onus is on me to give a problem that forces you to think. #TODO Write a question paper that is ChatGPT proof. Exploring AI could be a ToK subject. “How to interact with an AI?” We need a manual on how to use AI. Like Simon Willison says Content doesn’t suffice. You need pedegogy. What to serve you at what time, how, how to assess. Lots of businesses are filling this gap Students get great confidence when a teacher points to online content and says, I"ll tell you WHAT to see" and COMPLEMENTS that in their class “The map is not the territory.” Most people confuse sample mean for the actual. #ASK Parameter estimation -> Signal estimation -> State estimation Stats vs DL differ in that There is no notion of a defined “truth”. Hence reliability is not measurable Parameters have no value. Hence interpretability is ignored. #TODO Read 2020 National Education Policy. It’s quite modern. We need a manual on self-learning too Listening is not learning. You know only if you implement. Levels for students: I can solve it. I can explain why it works. I can find alternatives. I can apply it to a new area, reformulating (requires imagination.) For teachers, you also need: Responsible learning (extra careful about what to teach and how to teach, to exceite them, to teach at THEIR level). Show the universality and connecting to other concepts. E.g. noise reduction with FT is like using water to remove dirt. Transform to water domain, remove dirt, transform back to air domain. It’s better than dusting clothes to remove clothes. Washing machine programs are just different models of removing noise in the water domain. Teach people who WANT to learn AND who will APPLY it long-term. That’s what maximizes impact Grad students are more satisfying that way. Else, it is WASTED effort. (Not that it’s a bad thing for the student, but the effort IS wasted for the teacher) Therefore, I believe students should have general engineering first, and let students pick specialization later. Some universitie are doing that. #THINK Students remember my philosophy more than my content. We impart character, not just knowledge. Astrology and horoscopes serve a different function. They provide explainability, not predictive ability. As the world becomes less explainable, the need for astrology will grow. Explainability is about creating STORIES that fits data plausibly. It has nothing to do with data or truth. Explainability and predictive ability and reproducibility are all different. Maybe, Science is about the latter two, less about explainability. Astrology is a model. The map is not the territory. It’s an explanatory, not a predictive model. #THINK Therefore, my lessons are just explanations. Stats about experiments are STILL explanations. They are NOT reproducible or predictive. Hence not yet science The meaning of our life is the transformation we undergo in our lives #TODO Read “The Journey of Souls” by Michael Newton. A hypnotherapist #TODO Try regression therapy / hypnosis. Record it and listen to it. Just for fun! Rohini Deshpande Slam book was the Facebook of the 1900s Prepared mind is an extremely powerful tool for learning. Practice prepared mind When women drop out of education or career, that is also a waste from the teacher and system perspective The time for career growth is the same as child bearing time for women. That’s not true for men. But child rearing can be done by either. That’s not recognised. It’s 0K for a man to raise the child and make the home and 0K to treat that as the default Since men are more senior, it’s usually logical for them to stay in their jobs. That’s a systematic bias. When seniors advise women to step back. they respect it. That widens the barrier. Why not eliminate that situation? Be proud of the working women in the family Stats are just a symptom. They don’t explain the cause. (Map is not the territory.) Explanations are what really helps us fix the cause. Hence stories are important. Read Tinker Tailor Soldier Spy RV Athimber health tips: Eat foods with low glycemic index Eliminate free salt completely Voyage AI Embeddings have a higher quality, similar price compared to OpenAI embeddings. There’s a clear benefit to replacing text-embedding-3-large with voyage-3-lite. There’s a 200 MTok free tier currently. mixtral-offloading cleverly loads only the model layer required at any point, letting you run Mixtral 8x7b on Colab Free and on 16GB GPUs. This notebook runs on Colab Free too. CodeGPT is an alternative to Github Copilot that can use any LLM.