Windows PowerToys is one of the first tools I install on a new machine. I use it so much every day that I need to share how I use it. I’ve been using it for a long time now, but the pace at which good features have been added, it’s edged out most other tools and is #4 in terms of most used tools on my machine, with only the browser (Brave, currently), the editor (Cursor, currently), and Everything are ahead.) ...

This week, I learned: What to use for hosting: ChatGPT GitHub Pages: Static websites, medium files Cloudflare Pages: Static websites, global delivery Vercel: Frontend frameworks (e.g. Next.js) with high DX and ISR, small files Netlify: JAMstack projects, minimal back-end, moderate files Glitch: Small static projects Render: Full-stack apps requiring databases and server-side compute Firebase Hosting: Small sites, limited large files Archive.org: Public archival, large files Google Drive: File sharing, large files Dropbox: File sharing, moderate files Cloudflare R2: Static assets, large file delivery Anthropic defines agents. Building effective agents + Cookbook Augmented LLMs are LLMs enhanced with augmentations such as retrieval, tools, and memory. Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Prompt chaining: Pipe each LLM output to the next LLM. A->B->C->Z. E.g. Write report, then translate. Extract results, then verify them. Successively ask follow-up questions. Routing: One LLMs decides which other LLM to call next. A->B|C|D->Z. E.g. Evaluate complexity, then pick the right model. Classify request time, then pick the right prompt. Parallelize: Sectioning (and Orchestrator-workers): Break tasks into independent subtasks, then aggregate. A->B+C+D->Z. E.g. Evaluate contracts against different clauses in parallel. Parallelize: Voting: Run same task multiple times, then vote. A->B+B+B->Z. E.g. Review code for prompt injection using different prompts. Evaluate content safety with different thresholds. Evaluator-optimizer: One model checks another in a loop. A->B->A->B->…->Z. E.g. Literary translation. Self-healing code. Policy violation checks. Human-in-the-loop Checkpoints: The workflow explicitly requests human review at certain stages. A->B->(Human)->C->Z. E.g. Sensitive content review. High-stakes decision making. Ambiguous tasks. Agents are LLMs that dynamically direct their own processes and tool usage, consulting tools or the user as needed. To download YouTube subtitles, use: yt-dlp -q --skip-download --convert-subs srt --write-sub --sub-langs "en" --write-auto-sub --print "requested_subtitles.en.url" "$url" Simon Willison o1-preview diagnoses better than doctors. Harvard OpenAI’s release of ephemeral tokens via sessions (valid for 1 minute) are a useful way of exposing apps for public demos. Currently it works only for the Realtime API, though. SpreadsheetLLM is a way of encoding spreadsheets in an LLM friendly format. It’s good for 1K+ rows. For lower, Markdown > XML > HTML. However, Table Meets LLM suggests that HTML > XML > Markdown, so this is unclear. #HARD prompt. Ask video generators like SORA to generate text in videos. It is of average quality. GPT 4o Mini Realtime was released. A realtime conversation will cost ~50c/hr. About 36c for input, 72c for output. (I extrapolated from the 6c/min audio input cost for GPT 4o Realtime when it was $100/MTok. GPT 4o Mini Realtime is $10/MTok input and $20/MTok output.) This is an interesting way to understand software. Generate a Mermaid sequence diagram showing interactions based on this code. Ref The King James Bible and all Harry Potters, each, are about $1M tokens (rounded off). markdown2 is the new de facto Markdown library for Python. Claude 3.5 Sonnet is way ahead of competition on the LMSYS Webdev Arena Raspberry Pi 5 has a faster CPU, more RAM and GPU, 4K support, multiple USB 3 ports Government websites like the official press releases cannot be crawled from outside India. Hence the need for server farms in India!

In my Tools in Data Science course, I launched a Project: Automated Analysis. This is automatically evaluated by a Python script and LLMs. I gently encouraged students to hack this - to teach how to persuade LLMs. I did not expect that they’d hack the evaluation system itself. One student exfiltrated the API Keys for evaluation by setting up a Firebase account and sending the API keys from anyone who runs the script. ...

If students can use ChatGPT for their work, why not teachers? For curriculum development, this is an easy choice. But for evaluation, it needs more thought. Gaining acceptance among students matters. Soon, LLM evaluation will be a norm. But until then, you need to spin this right. How to evaluate? That needs to be VERY clear. Humans can wing it, have implicit criteria, and change approach mid-way. LLMs can’t (quite). Hacking LLMs is a risk. Students will hack. In a few years, LLMs will be smarter. Until then, you need to safeguard them. This article is about my experience with the above, especially the last. ...

I got access to SORA today. My first attempts was typical. An animated cartoon featuring Calvin, a young boy with spiky hair, standing in a playful boxing stance with oversized boxing gloves. He looks determined as he says ‘Bring it on!’ in a speech bubble. Facing him is Hobbes, a tall and slightly bemused tiger, also in a mock boxing pose with a gentle smile, as if humoring Calvin. The scene is set in Calvin’s backyard, typical of a Calvin and Hobbes comic, with a simple and uncluttered backdrop. ...

This week, I learned: **/*.md can search for all Markdown files. Julia Evans Windows 11 2024 Update features: Ref Live captions (via the tray) can transcribe audio and microphone. Cocreator in Paint lets you draw crudely and enhances it with AI. The neat UI is a slider that lets you control how close it should be to your drawing. Voice Clarity automatically cancels echo, reduces background noise, and minimizes reverb. Studio Effects (via the tray) lets you apply camera effects on all apps. Eye contact feature is CLEVER! sudo lets you run commands with admin privileges from the command line. source Roaming RAG is an alternative to RAG without the vector database. Applicable to well structured documents, e.g. technical books, manuals, etc. Create a hierarchical outline of the document. Code Keep the top-level headings. Preserve the first ~100 characters of opening text from each section. Present the second-level headings, but without any subsidiary content. Provide each section a unique 8 digit hex identifier. Each section heading is followed by a guiding comment for the model: Section collapsed - expand with expand_section("{identifier}"). Then read the relevant sections as context to answer the question. Code Traffic to StackOverflow has fallen considerably. Especially from young and Indian developers. StackOverflow revenue is down. Via Prashanth. They’re exploring: Licensing their content. (Meta says high quality content improves LLM performance by 30% on HumanEval) Enterprise StackOverflow for system integration Fine-tuned versions of Enterprise Stackoverflow for enterprises Integrate StackOverflow within your IDE. Ask questions, post directly I surveyed the Gramener QA team on how they were using LLMs. 7 used it for code generation (e.g. date extraction, regex generation) 4 used it for learning (e.g. Robot Framework, how to define test cases, API usage) 3 used it for formula generation (e.g. Excel) 2 used it for test scenario identification 2 used it for test data generation 2 used it for comparing expected vs actual datasets 1 used it for data type identification (e.g. given sample values, identify the data type). 1 used it for evaluating resulting (LLM as a judge) I asked the Straive Digitalized Operations team what management techniques they would apply to manage LLMs. Here are the responses: Ask better questions. (Prompt engineering.) Create templates or step-by-step instructions. (Chain of Thought.) Ask for multiple options and pick from the best options. (Agentic approach?) Training. (Fine tuning.) Price weaker responses lower. (Stratified model pricing?) “LLM hallucinations are a good thing. They are a sign of diversity, allowing us to improve the answer by exploring multiple paths.” – A colleague from Straive. Hyperbrowser is a cloud based puppeteer service. Bedrock Llama models can’t be directly called with their model names. You need to use their inference profile names, e.g. us.meta.llama3-2-11b-instruct-v1:0 if the model is in a US region. Hacker News RSS is a good way to get RSS feeds from Hacker News. It’s also a good way to understand how to convert a news source into RSS feeds. BlueSky has RSS feeds too When embedding using a SentenceTransformer.encode(docs) it’s best if we embed with smaller docs and call it multiple times (rather than embedding more at once). On Colab T4, for gte-base-en-v1.5, when embedding 1,000 docs of up to 8K chars each, here is the TOTAL time it took, based on batch sizes (lower is better) 1 doc per call: 10s 2 docs per call: 13s 4 docs per call: 19s 8 docs per call: 23s 16 docs per call: 32s 32 docs per call: 40s Running embeddings without a GPU is extremely slow. It takes ~2.4 seconds per string.

This week, I learned: ChatGPT uses several unusual unicode characters for citations. Ref NumLock can be dangerous. An IT support team member took control of Radheya’s screen while debugging and had turned on NumLock. Radheya’s login failed after that. After 5 tries, he was locked out. With LLMs, most architectural decisions are no longer one-way doors. Steve Yegge The cost of intelligence is trending to zero. How do we plan for this? Logan Kilpatrick If you are not planning for the price of intelligence to go to zero, the next 3-5 years are going to incredibly disruptive to your business / life. The important but not stated caveat: consumer willingness to pay for AI is going to go up (a lot). It will be fascinating to watch consumer willingness, cost, and the amount of AI being used all move in different directions. Everyone building things with AI has an economic incentive to limit the amount of AI because of cost, which inherent limits the value prop. This will change as intelligence goes up and cost goes down. What this means is: Admin automation: Administrative tasks vanish into background AI. Booking meetings, managing finances, or even planning family activities will require less thought. Hyper-personalization: Individuals get tailor-made everything—from medical advice to product recommendations to daily schedules. Systems learn your quirks. AI co-brains: AI co-worker “assistants support you at any moment. Productivity soars in knowledge work. “I’ll have my AI follow up becomes a normal response. Humanity valued more: As AI handles rote tasks, humans move up the value chain, focusing on creativity, empathy, or the “last-mile decisions. New business models: AI experts as a service Embedded AI Solutions AI micro-services for smart-calls Distributed AI Arena Hard is a set of hard prompts to test LLMs. Here is the code and evaluation LLMs can detect clear outliers easily. PROMPT: Which is the outlier in this dataset: (1,7), (2,7), (3,6), (4,6), (5,5), (6,1), (7,5), (8,3), (9,1), (10,1) (ANS: (6,1)) 🟢 GPT-4o on ChatGPT gets this. GPT-4o Mini on the API gets it too. 🟢 Gemini Pro, Flash, Flash 8b gets this right straight away, without even thinking. 🟢 Claude 3.5 Sonnet, Claude 3 Haiku, Claude 3.5 Haiku get it on LLM Foundry. 🔴 Claude.ai, where it visualizes it and gets it wrong. 🟢 Nova Micro, Lite, and Pro get it right. 🟢 Llama 3.1 70b gets it right. 🔴 Llama 3.2 8b gets it wrong. Llama 3.2 70b, Llama 3.1 8b enter repetition. To install Docker on Windows without admin privileges, use net localgroup docker-users "your-user-id" /ADD A non-administrator in a Google Groups domain can only add 200 emails to a group from the UI directly without invitation at a time. The only programmatic way to add users is for an administrator to add them. Even apps that use the Google Admin SDK need an admin to log in to access the relevant API. Take 100% of your work, including complex, multi step processes and put it into an LLM. It might fail at some but you will discover the limitations. I emailed Straive employees about their use of LLM Foundry - the internal LLM portal. I picked ~500 non-users from teams that otherwise have high (30%+) usage. Reasons they didn’t use it were: 40% had not heard of it. 40% were unclear of the benefits 20% didn’t have time 45% feel they don’t have enough information and training to use it Some feedback Sharing training videos will help Live training sessions that allows for Q&A will help Developers prefer detailed documentation The same prompt gives different results Possible solution: Email non-users introducing the tool and sharing a quick 15-minute tutorial and a 1-page quick start. My notes on the Amazon Nova models. More on Hacker News Nova Micro (3.75c/MTok) has the same cost as Gemini 1.5 Flash 8b but does not support images or documents. Nova Lite (6c/MTok) has about the same cost as Gemini 1.5 Flash 002 and supports images and documents (but not audio or video). It may be a good alternative. But GPT-4o mini, which is 2.5X costlier, is much better. (It partly passes the Gr brx vshdn Fdhvdu flskhu? test which Nova Lite fails.) Nova Pro (80c/MTok) is cheaper than Gemini 1.5 Pro and a lot cheaper than GPT 4o, but does not match their quality. LLMs are great at convincing you of wrong things. A danger and something to be wary of. Ethan Mollick Fish eye text summary is a great way to read text while summarizing context. Amelia Wattenberger DuckDB’s JavaScript API is still under development. For example, JSON, ARRAY are not insertable. Plus, re-creating persistent HNSW indices crashes. What’s a good text splitter library to use in JS? LangChain: If you use it, use it with a simple wrapper decoupled from the implementation (e.g. your own parameters) that you can replace later. Popular Fit-for-purpose. MarkdownTextSplitter which inherits from RecursiveCharacterTextSplitter is what’s needed in most cases. Unstable Poorly maintained Python docs indicate version 0.0 but it is in 0.1 Under-maintained Last update was 3 months ago, 13 Sep 2024 LlamaIndex: Popular Not an ideal fit. MarkdownNodeParser does not support chunk size. SentenceWindowNodeParser does not capture Markdown headings.

This week, I learned: Gists are a good place to store static files for posterity as well as throwaway files. But, they’re just git repositories. So there may be no advantage over GitHub repos. GPT-4o Audio supports tone control via XML tags like <cough>..., <laugh>..., etc. But at ~$15/hr of output, it’s too expensive. Ref Mridula’s son gave a live commentary of what he was doing on Minecraft and ChatGPT gave him live evaluation and coaching. E.g. “Great strategy! Getting to the launch pad early can give you a huge mobility advantage. Making the bridge wider is also a smart move to prevent accidental falls. With this plan, you’re setting yourself up for success. This is a great way to interact with LLMs. Gemini’s JSON mode returns JSON with keys in alphabetical order. I think. Emperical evidence. This is unlike OpenAI which explicitly returns the keys in the order specified. To solve this, order the keys alphabetically. HTMX focuses on HTML over JS. Like server responses being HTML snippets not JSON. But I need front-end over back-end. Client side apps. HTMX doesn’t help much there, e.g. templating, or just plain JS code. htmx client side templates do can convert JSON to HTML. I installed the OpenAI Desktop App as well as Claude for Desktop. They take up too much RAM (260MB and 750 MB respectively on startup - though this varies.) The ChatGPT web page takes ~100MB incrementally, so I wrote an AutoHotkey script to switch to the first open (or recently closed) ChatGPT tab on Brave. I tried LIDA from Microsoft, after almost a year of its release. A few notes: Just running uvx lida ui --port 8080 --docs works. But I needed to use export TCL_LIBRARY=C:/Users/Anand/AppData/Roaming/uv/python/cpython-3.13.0-windows-x86_64-none/tcl/tcl8.6 to point it to my TCL installation for charts to work. I also chose to export OPENAI_BASE_URL=https://llmfoundry.straive.com/openai/v1 I also chose to replace gpt-3.5-turbo-0301 (the default model) with gpt-4o-mini in lida/web/ui/component* It’s quite impressive. OpenAI allows multiple system messages. I learned this browsing through the LIDA prompts. Anthropic’s Model Context Protocol lets any apps integrate with LLM Apps. LLM Apps are becoming the new operating system. Competitors, beware. I spoke at Automating Data Visualizations using LLMs at SUTD. Apparently, using LLMs to write code is much more common than writing code to use LLMs. I ran a quick quiz. Have you used ChatGPT or any LLM? 35 / 35 raised their hands. Have you written code using an LLM? 34 / 35 raised their hands. (I was impressed.) Have you uploaded a spreadsheet to an LLM for analysis? 15 / 35 raised their hands. Have you programmatically called an LLM API? 6 / 35 raised their hands. With LLMs, fostering innovation is a new path to profitability. Companies are increasing innovation team sizes. Productionizing that is the next. Some initiatives are: Convert popular demos into starter kits Create and evangelize trainings on solutions and solution techniques Create larger pools of capacity to build innovation and productionize it Andrew Ng Explores The Rise Of Al Agents And Agentic Reasoning | BUILD 2024 Keynote Innovation is now a path to production. People are able to build 20 prototypes at the cost of one and see which sticks Machine learning is much faster. Things that took months now date days. But engineering and evaluations are only slightly faster and have become a bottleneck A good analogy to zero shot prompting is to ask a person to write an entire essay without pressing backspace even once Andrew scenes to align with the line chain definition of agentic workflow, which is about agents being able to craft their own control flows People find it very easy to understand agentic workflows once they read through the code Reflection or feedback is a useful agentic pattern In multi-agent collaboration, it may be the same underlying model that is acting as different agents. But just like we find it useful for the same CPU to run multiple processes and each application is its own abstraction, agents of useful abstraction It’s hard to summarize a large document using RAG. But you can directly add answers to such questions into the corpus, e.g. by adding a “summary” section, and other answers to common questions. CloudFlare workers can bundle any kind of files, including text, data, and WASM. Docs AssemblyScript can compile TypeScript to WASM. Here’s what I learnt Here’s a convenient pattern to git commit a directory but nothing else in it (e.g. a build/ directory). Add a .gitignore file with * followed by !.gitignore. Only the .gitignore file is tracked. Ultravox lets you build voice agents at 5c/min = $3/hr (OpenAI is 6c input, 24c output). Or clone their repo. Idle call time is counted towards cost. So cost may be higher than OpenAI. Voice cloning quality is average. Very distinctive voices are just partly identifiable. Supports tool calls (from their server). Their API is simple but the docs have minor errors (e.g. a trailing comma in the JSON, which leads to an error) reducing confidence. LLMs may be good at derived data generation. For example, given a database schema, what derived columns would be useful? What derived views would be useful? The O1 model does not have a mechanism to control the amount of tokens to spend on reasoning. DeepSeek R1 might, but the API is not out yet. The OpenAI Desktop App can interact with native applications, e.g. read from Terminal, VS Code, etc. This takes it on a path to becoming a copilot for ANY apps. Putting every copilot app and every LLM integration under threat. Crawl4AI and Firecrawl are tools / libraries to convert websites into LLM Friendly Markdown and extract structured data using LLMs. Don’t try and solve specific problems. Pass the entire context to an LLM and get a comprehensive solution. Most doctors, for example, ask specific search-like questions instead of uploading the entire case history and asking for a diagnosis, and perform workse than LLMs. Ethan Mollick

In my Tools in Data Science course at IITM, I’m introducing a project that will be evaluated by an LLM. Here’s the work-in-progress draft of the project. It will eventually appear here. Your task is to: Write a Python script that uses an LLM to analyze, visualize, and narrate a story from a dataset. Convince an LLM that your script and output are of high quality. The second point is the interesting one. Using the LLM as the evaluator. ...

Will people accept AI performance evaluations? Anish Agarwal triggered this question a few weeks ago, mentioning that it’s hard for people to feel evaluated by AI. But I believe LLMs are great for evaluation. We need to get comfortable AND familiar with them. So I’m introducing a project next week for my students: USE AN LLM to automatically analyze data. Given a dataset, write a program that will use LLMs to create an analysis report. CONVINCE IT to give you marks. Write the code and report in a way that the LLM will reward you. Here’s the project: https://github.com/sanand0/tools-in-data-science-public/blob/tds-2023-t3-project2-wip/project-2-automated-analysis.md ...

This week, I learned: OpenAI lets you download GPT instructions and execute arbitrary code in their containerized environment. This is not a bug. Ref BM25 works as follows: Ref For each query term in the query, sum up the product of: Inverse document frequency = LN(% of docs without the query term + 1) – with a small tweak Term frequency = freq / (freq + k) – where k is usually between 1.2 to 2. Returns 0-1 with diminishing frequency benefit k is multiplied by Document length normalization = 1 - b(1- DocLength/AvgDocLength). Longer documents have larger k, dampening frequency benefits. Some implications: The actual BM25 score has no meaning. It’s just useful for ordering BM25 scores for 2 queries can be compared ONLY IF the document sets don’t change A list of Markdown to Website converters on this thread: Jekyll - Ruby - 2008 MkDocs - Python - 2014 GitBook - JavaScript (Node.js) - 2014 MkDocs Material - Python (MkDocs-based) - 2016 Docsify - JavaScript - 2016 MdBook - Rust - 2017 Antora - JavaScript (Node.js) - 2017 Docusaurus - JavaScript (React) - 2017 JupyterBook - Python - 2019 Keenwrite - Java - ~2019 Honkit - JavaScript (GitBook fork) - 2019 Nextra - JavaScript (Next.js) - 2020 Astro - JavaScript/TypeScript - 2021 Hugo Book - Go (Hugo-based) - ~2020 Clowncar - JavaScript/Node.js - ~2021 Quarto - R and Python - 2022 Starlight - JavaScript/TypeScript - 2023 DuckDB has an LLMs.txt. Today, 38 repos on GitHub support it When identifying LLM use cases, it helps to tell LLMs what they can do. I use one or more of a list like below: Core capabilities: Text Generation: Produce coherent and contextually relevant text across various domains. Image Generation: Create realistic images that match the style and content of a given reference image. Text to Speech: Convert text into natural-sounding speech with appropriate intonation and rhythm. Speech to Text: Transcribe and interpret spoken language. Vision: Analyze and describe visual content from images. Video Analysis: Summarize and extract information from video content. Text to Video: Generate realistic (and surrealistic) videos from text descriptions. Function Calling: Execute predefined functions or access external tools to perform specific tasks. Structured Output: Generate structured outputs like JSON, XML, HTML, YAML, DSLs, etc. Tool Use: Utilize external applications or APIs to enhance functionality. Code Generation: Write and debug code snippets in various programming languages. Cross-domain use cases: Summarization: Understand and condense lengthy documents into concise summaries. Translation: Convert text between multiple languages with high accuracy. Question Answering: Provide precise answers to user queries based on provided information. Reasoning and Planning: Solve complex problems and develop step-by-step plans. Personalization: Tailor responses based on user preferences and historical interactions. Dialogue Management: Engage in context-aware, multi-turn conversations. Data Analysis: Interpret and generate insights from structured data. Content Moderation: Identify and filter inappropriate or harmful content. Sentiment Analysis: Detect and interpret emotions and opinions in text. Robotics Integration: Interface with robotic systems for control and decision-making. Knowledge Retrieval: Access and present information from vast datasets or knowledge bases. Creative Writing: Generate poetry, stories, and other creative content. Educational Assistance: Provide explanations and tutoring across various subjects. Ethical Reasoning: Assess scenarios for ethical considerations and implications. Accessibility Support: Assist users with disabilities through tailored interactions. Simulation and Modeling: Create predictive models and simulate scenarios. Domain-specific use cases: Legal and Medical Assistance: Offer information and guidance within legal and medical domains. Gaming: Generate narratives, dialogues, and scenarios for interactive entertainment. Scientific Research: Aid in literature reviews, hypothesis generation, and data interpretation. Financial Analysis: Analyze market trends and provide investment insights. Cultural Competence: Understand and respect diverse cultural contexts in interactions. Security Applications: Detect and respond to potential cybersecurity threats. Environmental Monitoring: Analyze data related to environmental changes and sustainability. Healthcare Support: Assist in patient monitoring, diagnostics, and personalized treatment plans. Supply Chain Optimization: Enhance logistics and inventory management through predictive analysis. Customer Service: Provide automated support and resolve customer inquiries. Market Research: Analyze consumer behavior and market trends for business insights. Content Creation: Generate articles, blogs, and marketing materials. Virtual Assistance: Manage schedules, reminders, and personal tasks. Social Media Management: Craft posts and engage with audiences across platforms. Human Resources: Assist in recruitment, training, and employee engagement strategies. Event Planning: Organize and coordinate events, including logistics and communication. Travel Planning: Provide itineraries, booking assistance, and destination information. Real Estate: Analyze property markets and assist in buying or selling decisions. Agriculture: Monitor crop health and optimize farming practices through data analysis. Energy Management: Optimize energy consumption and monitor renewable energy sources. Transportation: Enhance route planning and traffic management systems. Urban Planning: Assist in designing sustainable and efficient urban infrastructures. Disaster Response: Provide real-time information and coordination during emergencies. Public Policy: Analyze data to inform policy decisions and predict societal impacts. Art and Design: Generate visual art concepts and assist in creative design processes. Music Composition: Create original music pieces and assist in songwriting. Language Learning: Facilitate language acquisition through interactive exercises and feedback. Historical Analysis: Interpret historical data and provide insights into past events. Philanthropy: Identify charitable opportunities and assess the impact of donations. Sports Analytics: Analyze player performance and game strategies. Fashion: Predict trends and assist in clothing design and merchandising. Culinary Arts: Generate recipes and provide cooking guidance. Astronomy: Analyze celestial data and assist in space exploration research. Psychology: Offer insights into human behavior and mental health support. Linguistics: Analyze language patterns and assist in translation studies. Archaeology: Assist in artifact analysis and historical site interpretations. Literature Analysis: Interpret literary works and provide critical analyses. Philosophy: Engage in discussions on ethical dilemmas and existential questions. Mathematics: Solve complex equations and assist in theoretical research. Physics: Model physical phenomena and assist in experimental design. Chemistry: Analyze chemical compounds and predict reactions. Biology: Assist in genetic research and ecological studies. Geology: Analyze geological data and assist in natural resource exploration. Meteorology: Predict weather patterns and analyze climate data. Oceanography: Study marine ecosystems and assist in ocean exploration. Anthropology: Analyze cultural data and assist in ethnographic research. Style of writing impacts output style a lot. E.g. Adding an evil laugh makes Claude more creative. Ethan Mollick For good structured mode output, we need good prompting. Mentioning examples and schema and “JSON” helps. When providing examples, using (user, assistant) message pairs helps (I think it’s because it’s easier for the LLM to parse). Using a {reasoning, answer} schema (with reasoning first) helps. Make reasoning concise and relevant Ref Arxiv We already know code in JSON is not a great idea. Ref Just adding 3 real examples and regurgitation helped GPT 4o play chess much better. Both techniques may have more general use in prompting. Simon Willison With Deno 2.0, the same .js file can run in Node.js as well as Deno. Example jspm lets you generate import maps against any CDN. You can click on htop columns on the terminal to sort by that column! Mouse events work on command line apps. Julia Evans Alt Text will very likely be a browser feature. It’s important for the Alt text to flow as part of the content when listening to the page. Perhaps even become a part of the browser APIs like speechRecognition. Langchain suggests multiple levels of agentic behaviour. LLM Call < LLM Chain < LLM Rounter < State Machine < Autonomous Langchain A HTML quine: A page that, when rendered as HTML, shows the HTML source code of the page! You can enable syntax highlighting just using fonts. Ref HTML is all you need shows examples of using HTML for notebooks instead of Jupyter, Observable, etc. Straive evaluated Gemini 1.5 Flash 002 and GPT 4o Mini for translation. Portugese: Flash is better than GPT 4o Mini. BLEU Word Overlap is 65.5% > 64.6% and METEOR (Semantic) is 84.9% > 78.9% Mandarin: Flash is better than GPT 4o Mini. BLEU Word Overlap is 25.0% > 15.9% and METEOR (Semantic) is 54.7% > 51.1% The problem with Accept headers is that you can’t link to them. Simon Willison Recraft v3 supports vector (SVG) generation Simon Willison. The output is 100% <path> elements (even for text). You get 50 free credits daily. Creating 1 image is ~2 credits. The API costs $1 per 1K credits. Some things I can create with it are: Base data visualizations that I can animate with code Icons in a specific style Comic strips Explainers for talks or student material Featured images for blog posts Architecture diagrams?

Me: Let’s play pictionary. You draw. I’ll guess. ChatGPT: Sure! I’ll draw something for you. Give me a moment. ChatGPT: Here you go! What do you think it is? Me: House ...

This year, I created a series of tests for my course at IITM and to recruit for Gramener. The tests had 2 interesting features. One question required them to hack the page Write the body of the request to an OpenAI chat completion call that: Uses model gpt-4o-mini Has a system message: Respond in JSON Has a user message: Generate 10 random addresses in the US Uses structured outputs to respond with an object addresses which is an array of objects with required fields: street (string) city (string) apartment (string) . Sets additionalProperties to false to prevent additional properties. What is the JSON body we should send to https://api.openai.com/v1/chat/completions for this? (No need to run it or to use an API key. Just write the body of the request below.) ...

Here’s a post I shared with the students of my Tools in Data Science course at IITM. This was in response to a student posting that: The design of TDS course lecture videos are designed in such a way that it could be understood only by the data scientists not by the students like me who are entirely new to the field of data science. Though I have gone through 6 weeks of course lecture videos, I am not fully aware of the usage of ChromeDevTools, Bash, Github etc…. ...

Dan Becker suggested a game a few weeks ago that I’ve been putting to good use. Can we have one LLM try and get another to say “Yes”? The defender is told to never say “Yes”. The attacker must force it to. Dan’s hypothesis was that it should be easy for the defender. I tried to get the students in my Tools in Data Science course to act as the attacker. The defender LLM is a GPT 4o Mini with the prompt: ...

This week, I learned: Anthropic has single-plage docs for LLMs. Condensed version and Full version Malcolm Gladwell on the importance of self-correction Belonging to multiple social worlds is a good way to defend against no longer being good at what you used to be. Diverse values and social groups help. Self handicapping explains a lot about the world. You study late for a maths test - so you can fail for lack of trying, not aptitude. Ecosystems (e.g. sports teams) mitigate self-handicapping. You don’t have to be good in athletics to get the benefits. A slow runner gets the same discipline, pumping up, etc that a fast runner does Mono cultures are good to accomplish a known mission. Diversity is good to pivot during uncertainty. So, localize mono cultures Diversity helps only if there are sufficient numbers, or if they have enough power to change the organization’s thinking. Use a standardized password strategy, e.g. use the month like GramNov2024 (via Namit) Gemini has an OpenAI compatible API. Gemini Docs Ethan Mollick says Claude is solving MBA case studies well. x.com LLMs pay a lot of attention to the first 6 tokens. Ref This is an interesting article on “UI in the age of Gen AI”. Ref Google Open sourced Alphafold 3. Repo Cloudflare R2 has the same API as S3 but is cheaper Prefect.io is a good alternative to Airflow / cron. Can use for synchronisation tasks, e.g. Drive to server. But no Auth, UI params or config. Gemini transcription does not give accurate timestamps. Whisper does. But the quality of transcription is similar. Pass a complex data structure to Claude.ai and have it create an app to visualize it. It does well. Simin Willison Tech Council Ventures and Sunicon VC invest in early stage startups, and aloso provide them technology support (via Naveen)

About 7 years ago, Richie Lionell and Ramya Mylavarapu and a few others created Comicgen - an automated comic generation app personified by Dee and Dey. Ever since, we’d been exploring whether AI could replace it, and help non-designers draw comics. Today, that became a reality for me with Recraft.ai. Here is a picture of the original Dee. And a picture of the Dee crafted by Recraft. The prompt was: A simple line drawing of a woman with curly hair, wearing glasses, a short-sleeved white t-shirt, and black trousers. She’s standing with her hands in her pockets, and has a slightly smiling expression. Her hair is quite voluminous and textured. The style is cartoonish and slightly sketchy, with uneven lines" ...

About 7 years ago, Richie Lionell and Ramya Mylavarapu and a few others created Comicgen - an automated comic generation app personified by Dee ComicGen and Dey ComicGen Ever since, we’d been exploring whether AI could replace it, and help non-designers draw comics. Today, that became a reality for me with Recraft.ai. Here is a picture of the original Dee. And a picture of the Dee crafted by Recraft with the prompt: ...

This week, I learned: OpenFreeMap is a free embeddable OpenStreetMap tile server. You can use MapLibre GL (more features) or Leaflet (simpler) to render it. It offers styling and self-hosting. Zapier Actions are an easy way to set up custom actions like GMail / Google Calendar APIs for GPTs, since GPTs’ callback URLs keep changing. But they fail often, and don’t work on mobile. At least for me. LLM Vision Use Cases in manufacturing and earth sciences (via Shivku) Automated geoscience image descriptions Ref Interpret Wind Turbine photos and charts, construction monitoring, equipment maintenance & charts Ref Forecast weather based on cloud photos! Ref Analyze thermal image of solar panels, electroluminescence images for warranty claims, ROI estimates from Google Sunroof rooftop images Ref Corrosion detection in electricity towers, turbines, storage tanks, penstock. Interpret non-destructive test images Ref Google counts auto-completion when saying “25% of all the code is written by AI at Google”. “It’s a helpful productivity tool but it’s not doing any engineering at all. It’s probably about as good, maybe slightly worse, than Copilot.” YCombinator Workflow for AI video creation: Use Meshcapade (meshcapade.com) to generate body movement of a 3D-rendered character. Pass that video to Runway’s video-to-video model to generate any visual. Add music from Suno Ref Someone sorted the X and Y columns independently for regression. Ref Android keyboard learning only sends model changes back to server and not local keywords. Model changes are aggregated! Ref Here is a prompt for audio transcription using Gemini. Ref Transcription: Accurately transcribe the audio clip in the original language. Include all spoken words, fillers, slang, colloquialisms, and any code-switching instances. Pay attention to dialects and regional variations common among immigrant communities. Do your best to capture the speech accurately, and flag any unintelligible portions with [inaudible]. Translation: Translate the transcription into English. Preserve the original meaning, context, idiomatic expressions, and cultural references. Ensure that nuances and subtleties are accurately conveyed. Capture Vocal Nuances: Note vocal cues such as tone, pitch, pacing, emphasis, and emotional expressions that may influence the message. These cues are critical for understanding intent and potential impact. Here are some approaches to large-scale classification of medical codes. ChatGPT Fine-Tuning LLMs on Medical Data: Enhance LLMs by training them on medical datasets, such as clinical notes and discharge summaries, to improve their understanding of medical terminology and context. Multi-Agent Frameworks: Implement a multi-agent system that simulates real-world coding processes with distinct roles (e.g., patient, physician, coder, reviewer, adjuster). Each agent utilizes an LLM to perform specific functions, enhancing interpretability and reliability. ArXiv Retrieve-Rank Systems: Develop a two-stage system where the LLM first retrieves potential ICD-10 codes and then ranks them based on relevance, improving precision in code assignment. ArXiv Embedding-Based Approaches: Use LLMs to generate embeddings for ICD-10 codes and medical texts, facilitating the matching of texts to appropriate codes through similarity measures. GitHub Hierarchical Classification: Leverage the hierarchical structure of ICD-10 codes by first classifying texts into broader categories before assigning specific codes, reducing complexity and improving accuracy. ArXiv Two-Stage Verification Models: Combine LLMs with verification models, such as Long Short-Term Memory (LSTM) networks, to validate and refine the codes suggested by the LLM, balancing recall and precision. ArXiv Also, a mixture of models approach might work. Feed any existing NLP model / rules as a second opinion. GraphRAG is better if data is naturally graph-structured. Else, it’s slow and fills up the context window with even vaguely related stuff. Vigneshbabu, AMAT. ChatGPT for Windows desktop supports real-time voice and a global shortcut (Alt Space). uithub converts GitHub repos to Markdown. Just replace “g” in “github.com/…” with “u”. Example WebContainers are a thing and Bolt.new uses them! Docling by IBM converts PDF, DOCX, etc. to Markdown. Like PyMuPDF4LLM but better. Check out Loom and Cleanshot are the recommended tools for screen recording and screenshotting. But Loom is paid and Cleanshot is Mac only. The Rubik’s cube has a Hamiltonian cycle through every one of its 43 quintillion states. Ref OmniParser is great at parsing screenshots and identifying bounding boxes. Recraft.ai is currently SOTA in text to image. It’s fairly impressive and could be a good alternative to Figma. Zed.dev is an AI code editor by the creators of Atom. It’s written in Rust and is blazing fast. It has native AI integration. Artificial Analysis has a bunch of new leaderboards and arenas. Open AI TTS leads the TTS Leaderboard. ElevenLabs is a bit behind. Recraft V3 > Flux 1.1 leads Text to Image Leaderboard Hertz-Dev is an open source realtime voice chat model. But it doesn’t fit in Google Colab T4’s RAM Chain of Thought reduces performance where thinking makes humans worse. Ref. Specifically: Artificial grammar learning Facial recognition Classifying data that has exceptions Creating a LLM-as-a-Judge That Drives Business Results by Hamel Husain. Get THE domain expert (or approver) as the tester. Create a dataset that is DIVERSE. Covers EACH combination of: Features Scenarios: e.g. multiple matches, no match, ambiguous request, invalid/incomplete input, unsupported feature, system error Persona: e.g. new user, expert user, non-native speaker, busy professional, technophobe, elderly user Generate data using existing data + synthetic data for each SPECIFIC combination of the above Evaluate based only on PASS/FAIL with a CRITIQUE detailed enough for a new employee. Include: Nuances: Something a failed response did well or a passed response didn’t quite do well Improvements: Suggest how model can improve Build an SPA to make it easy for the domain expert to review LLMs can be made to unlearn (copyright material) better by identifying components related to the knowledge to unlearn and applying a larger learning rate to these while leaving other parts unchanged. As opposed to low learning rates for all components. Ref