speech-to-text 7

Transcribe talk recordings with Q&A. Transcribe this talk. DO NOT MISS ANY PART OF THE TALK. Drop verbal tics and fillers (um, uh, etc). Correct spelling and grammar but otherwise don't modify the original words. Add English translations to any non-English parts. Mark inaudible or unclear segments as "[inaudible]". Mark uncertain words with like "[word?]" or ambiguous possibilities like "[word1? word2?]". Break it into LOGICAL paragraphs beginning with timestamps, e.g. "[00:13] When did ..." For audience questions, prefix with "**Question**: ..." and answers with "**Answer**: ..." **Make key points / takeaways / memorable statements bold**. <!-- #TODO List details of talk or share slides, for context --> If video is provided, add this line: ...

This week, I learned: There’s a very interesting HN discussion on the AI coding of CloudFlare Workers OAuth Provider. My takeaways: #ai-coding Write very comprehensive specs. Use LLM to create the specs. Reviewing is a skill we need to develop. Understanding others’ code takes effort. But LLM code is easier to review because it’s immediate and has no ego. Unit tests are critical. Use LLMs for well understood specs, APIs, platforms and libraries to really save time. Logic-less stuff like Markdown, JSON and HTML templates are a LOT easier to verify. Do more of that. We can only make so many decisions in a day. AI coding saves us that effort. Experts are not experts in every area. They benefit from LLMs in other areas. LLMs are great for rubber ducking. Speaking and speccing really help. LLMs make mistakes. So do most humans. LLM speed makes coding more exhausting. Use LLMs to understand codebases. AI coding could reduce demand for developers. E.g. Sysadmin demand plummeted with cloud infra and infrastructure-as-code. But, niche use cases could grow, like how demand for photographers grew despite point-and-shoot cameras. Transaction cost of hiring even 1 person is high and that will likely be a bottleneck. Plus people can use LLMs themselves, so that will dampen niche demand. Google Introduced Google Vids last year. It’s a video creator styled like PowerPoint. Looks promising. FastMCP looks like an easy way to build MCPs. (Yet to try it) O3 and to a lesser extent, Claude Sonnet 4, are the models that can accurately summarize complex subjects and create a list of links without hallucinations. Ref Claude Trace lets you record all interactions with Claude Code. Elevenlabs now supports emotion and interruption. Ref Thinking longer alone is not enough to scale intelligence. We need better models, too. Ref Indian High Court judgements are now available as a public dataset on AWS and updated periodically. Ref A few observations in AI code editors’ styles. O3 is better at finding bugs than Jules, which tends to try and fix them rather than discover them. Codex writes more minimal edits in PRs than Jules, which is more verbose. Claude Code remains the best at faithfully creating and updating front-end apps. Deep Research is great for fact-checking my notes! ChatGPT Web bench evaluates LLMs in web development. Claude Sonnet remains ahead. Vision language models heavily rely on past training and miss changes they don’t expect. Ref Pure CSS tooltips are possible. Julia Evans Google has an OAuth Playground which is a convenient way to get a temporary OAuth token. At the moment, the best speech to text for Android appears to be ChatGPT’s transcription. The default Android text to speech (which I thought was good) no longer feels adequate. Gemini mis-hears and doesn’t wait till I’m done. Whisper ASR has poor noise cancellation and a 30 second limit. anyascii is a better alternative to unidecode. It supports more characters and also supports transliteration. I use it to strip out non-ASCII in ChatGPT’s output. Commit DeepWiki creates docs for humans GitHub repos. Example. It’s verbose, human-facing, and does not understand the nuances of context and implications. Context7 creates llms.txt for LLMs. Example. It’s concise, example-oriented, and works only if there are code snippets relevant (e.g. API calls) that can be generated from the codebase. Like creating an llms.txt automatically, e.g. https://context7.com/textualize/textual/llms.txt #ai-coding We will move towards an organization structure where developers are embedded with business teams rather than working as a separate group. Sort of like embedded executive assistance instead of a central typing pool. Making AI Work

This week, I learned: OpenFreeMap is a free embeddable OpenStreetMap tile server. You can use MapLibre GL (more features) or Leaflet (simpler) to render it. It offers styling and self-hosting. Zapier Actions are an easy way to set up custom actions like GMail / Google Calendar APIs for GPTs, since GPTs’ callback URLs keep changing. But they fail often, and don’t work on mobile. At least for me. LLM Vision Use Cases in manufacturing and earth sciences (via Shivku) Automated geoscience image descriptions Ref Interpret Wind Turbine photos and charts, construction monitoring, equipment maintenance & charts Ref Forecast weather based on cloud photos! Ref Analyze thermal image of solar panels, electroluminescence images for warranty claims, ROI estimates from Google Sunroof rooftop images Ref Corrosion detection in electricity towers, turbines, storage tanks, penstock. Interpret non-destructive test images Ref Google counts auto-completion when saying “25% of all the code is written by AI at Google”. “It’s a helpful productivity tool but it’s not doing any engineering at all. It’s probably about as good, maybe slightly worse, than Copilot.” YCombinator Workflow for AI video creation: Use Meshcapade (meshcapade.com) to generate body movement of a 3D-rendered character. Pass that video to Runway’s video-to-video model to generate any visual. Add music from Suno Ref Someone sorted the X and Y columns independently for regression. Ref Android keyboard learning only sends model changes back to server and not local keywords. Model changes are aggregated! Ref Here is a prompt for audio transcription using Gemini. Ref Transcription: Accurately transcribe the audio clip in the original language. Include all spoken words, fillers, slang, colloquialisms, and any code-switching instances. Pay attention to dialects and regional variations common among immigrant communities. Do your best to capture the speech accurately, and flag any unintelligible portions with [inaudible]. Translation: Translate the transcription into English. Preserve the original meaning, context, idiomatic expressions, and cultural references. Ensure that nuances and subtleties are accurately conveyed. Capture Vocal Nuances: Note vocal cues such as tone, pitch, pacing, emphasis, and emotional expressions that may influence the message. These cues are critical for understanding intent and potential impact. Here are some approaches to large-scale classification of medical codes. ChatGPT Fine-Tuning LLMs on Medical Data: Enhance LLMs by training them on medical datasets, such as clinical notes and discharge summaries, to improve their understanding of medical terminology and context. Multi-Agent Frameworks: Implement a multi-agent system that simulates real-world coding processes with distinct roles (e.g., patient, physician, coder, reviewer, adjuster). Each agent utilizes an LLM to perform specific functions, enhancing interpretability and reliability. ArXiv Retrieve-Rank Systems: Develop a two-stage system where the LLM first retrieves potential ICD-10 codes and then ranks them based on relevance, improving precision in code assignment. ArXiv Embedding-Based Approaches: Use LLMs to generate embeddings for ICD-10 codes and medical texts, facilitating the matching of texts to appropriate codes through similarity measures. GitHub Hierarchical Classification: Leverage the hierarchical structure of ICD-10 codes by first classifying texts into broader categories before assigning specific codes, reducing complexity and improving accuracy. ArXiv Two-Stage Verification Models: Combine LLMs with verification models, such as Long Short-Term Memory (LSTM) networks, to validate and refine the codes suggested by the LLM, balancing recall and precision. ArXiv Also, a mixture of models approach might work. Feed any existing NLP model / rules as a second opinion. GraphRAG is better if data is naturally graph-structured. Else, it’s slow and fills up the context window with even vaguely related stuff. Vigneshbabu, AMAT. ChatGPT for Windows desktop supports real-time voice and a global shortcut (Alt Space). uithub converts GitHub repos to Markdown. Just replace “g” in “github.com/…” with “u”. Example WebContainers are a thing and Bolt.new uses them! Docling by IBM converts PDF, DOCX, etc. to Markdown. Like PyMuPDF4LLM but better. Check out Loom and Cleanshot are the recommended tools for screen recording and screenshotting. But Loom is paid and Cleanshot is Mac only. The Rubik’s cube has a Hamiltonian cycle through every one of its 43 quintillion states. Ref OmniParser is great at parsing screenshots and identifying bounding boxes. Recraft.ai is currently SOTA in text to image. It’s fairly impressive and could be a good alternative to Figma. Zed.dev is an AI code editor by the creators of Atom. It’s written in Rust and is blazing fast. It has native AI integration. Artificial Analysis has a bunch of new leaderboards and arenas. Open AI TTS leads the TTS Leaderboard. ElevenLabs is a bit behind. Recraft V3 > Flux 1.1 leads Text to Image Leaderboard Hertz-Dev is an open source realtime voice chat model. But it doesn’t fit in Google Colab T4’s RAM Chain of Thought reduces performance where thinking makes humans worse. Ref. Specifically: Artificial grammar learning Facial recognition Classifying data that has exceptions Creating a LLM-as-a-Judge That Drives Business Results by Hamel Husain. Get THE domain expert (or approver) as the tester. Create a dataset that is DIVERSE. Covers EACH combination of: Features Scenarios: e.g. multiple matches, no match, ambiguous request, invalid/incomplete input, unsupported feature, system error Persona: e.g. new user, expert user, non-native speaker, busy professional, technophobe, elderly user Generate data using existing data + synthetic data for each SPECIFIC combination of the above Evaluate based only on PASS/FAIL with a CRITIQUE detailed enough for a new employee. Include: Nuances: Something a failed response did well or a passed response didn’t quite do well Improvements: Suggest how model can improve Build an SPA to make it easy for the domain expert to review LLMs can be made to unlearn (copyright material) better by identifying components related to the knowledge to unlearn and applying a larger learning rate to these while leaving other parts unchanged. As opposed to low learning rates for all components. Ref

This week, I learned: DPO is a simpler alternative to RLHF for fine-tuning. Several HuggingFace models use DPO for training Name2Vec is a potential embedding for names. Google Knowledge Graph ID powers the Knowledge Graph. If it begins with /m/ it’s the same as the FreeBase ID. This is now available as WikiData. e.g https://www.wikidata.org/wiki/Property:P2671 I tried running Mixtral-8x7b locally (via Llamafile) and on together.ai. It’s good, but far from GPT 4. Generic computate-intensive algorithms eventually beat domain-specific tuning, because of Moore’s law. Ref The hidden brain podcast. the mystery of beauty Evolution drove us to beauty as an efficient survival mechanism. Understanding the world is one such mechanism. Hence we enjoy maths and chess ⭐ This leaderboard included paid models like GPT4 and Claude and compared them with open models on HUMAN + system benchmarks Lez Friedman Podcast: Jeff Bezos Build stuff that is is ubiquitous that other people take it for granted. The initial idea needs to be that obvious and easy. Like one click purchase or customer reviews Build stuff that other people can build on. Internet makes startups possible. Infrastructure is about enabling others at scale Decision making approaches: single person decides on two way doors. Deliberate as a team on one way doors Conflict resolution: disagree and COMMIT. NO sniping, I told you so, malicious compliance. Avoid compromise. Avoid decision by attrition (most persistent wins). People are inherently biased towards hierarchy. So the senior most person should speak last We have a happiness bias. Contracted by choosing the unhappier options first The map is not the territory. The metric is not the objective. We need metrics. But make sure you know why See the world through the eyes of the customer. Use your own product. It’s living their lives that makes customer obsession real. Jeff Bezos called their own customer care to see how long the actual wait time was. It was much longer than the metric reported How to prioritize. whatever problems customers will still face in 10 years are the big problems. These are worth putting time into because they are stable in time People working on big problems will never get down to the small problems. So have a dedicated team that works only on the paper cuts. It should be a dedicated team We co evolve with our tools. We build tools and then our tools change us. It reprograms our brains Cut out 10 minutes to the beginning of each meeting for people to read the material. They never reread anyway. This makes the meetings more productive Powerpoint is designed for persuasion, not truth seeking. It is also easier for the author than for the reader. Prefer narratives that are focused on finding the truth and are easier for the audience though tougher for the author ⭐ whisper-standalone-win provides a Windows binary for Faster-Whisper. It just needs CUDA and cuDNN installed. Then whisper-faster.exe video.mkv --language=English --model=medium generates the transcript. LLM use cases by Benedict Evans “Every text box on the internet will get an LLM” “Infinite interns” “Every UNIX function has become a company.” “Every ChatGPT suggestion…” llm360 publishes models along with training datasets. In The Age of AI has begun, Mar 2023, Bill Gates says, “In my lifetime, I’ve seen two demonstrations of technology that struck me as revolutionary.” The GUI (1980) and ChatGPT (2022). Rubeus is a HTTP proxy for multiple LLMs with load-balancing, fallbacks and retries. GPTRouter is a Python interface for multiple LLMs with fallbacks and retries. ⭐ Token Tally has an LLM Cost Tool that estimates GPU memory required and token cost across cloud providers.

The Grammer of Sound. (Free registration required.) Fast-Talk has created a product which can search sound files for text. So you can type “Dil” to find all Hindi songs having the word.

IBM has this cool tool: Multimodal browser extension. With this, I can create a textbox on my site, which you can fill in by speaking to the web page. The underlying standard is W3C’s XHTML+Voice. Two years ago, I’d have downloaded it and had it on my site in a few hours. Today, I’m a married consultant. I don’t have a few hours. :-(

I thought the next generation devices would use voice input. But maybe written is more likely. You can write on IBM’s Thinkpad and on Sony’s VAIO desktop.