anthropic on S Anand

Things I Learned - 24 May 2026

Sun, 24 May 2026 00:00:00 +0000

This week, I learned:

BitWarden seems to be sneakily jacking up prices and going towards a PE sale. Might be time to shift out or self host. Sigh, I just migrated into it… Source
Andrej Karpathy has joined Anthropic. Likely to use Claude to build better Claudes - automating AI research. Also, it probably isn’t a good time to build an AI education platform. Claude
The open-source Chinese models about 6 months behind frontier models. Qwen 3.7-Max is on par with Claude 4.5 Opus (Nov 2025) and Gemini 3 Flash (Dec 2025).
Google basically became Gemini. Entirely! I’m not sure there’s a difference any more. Which means it will scrape websites and not send traffic through - just killing the search economy. But it’s far more useful. Claude
I wanted a list of sites I log into with my Google Account. Google’s Linked apps page does that. Unfortunately, I can’t find a way to use Google Takeout to export that data. So I wrote a scraper which can be single-shot prompted these days.
As long as you remember to exhale, your chances of recovery from being ejected into space is pretty good for the first 15-60 seconds. Gemini
I don’t understand half the comments I read on LinkedIn. Earlier, I was able to separate good from bad. Now, I’m not sure if what I read is actually insight or idiocy. Is the AI use making their comments too smart or making my brain too dumb?
“Pax Memoriae”: peace of memory. Putting past conflicts to rest. The best part of it was, I learnt the phrase by typing “Pax” into VS Code and wasn’t sure what to write next. Before I could search for it, GitHub Copilot completed it. I searched for what it meant, and it was so apt!
Children’s vision is worse than adults, but filter less and absorb ore irrelevant information than adults. This is useful for learning and surprise detection, but costly for focus, speed, and relevance. ChatGPT
The word phobia comes from the Greek god of fear, Phobos, which is the name of one of Mars’ moon. Deimos, the other moon, is the Greek god of dread/terror. They’re the children of Ares (Mars), the god of war. Nice planet.
On WhatsApp, I can type @Meta AI and then /imagine to have it draw an image. The quality is OK - not great, not terrible.
Surprising but GPT Realtime Whisper ( new model) isn’t as good as the older open-source Whisper models. Also, Gemini 3 Flash Preview is as good at transcription as Gemini 3.1 Pro Preview for up to medium-length text. LLM Audio Transcription benchmark
Google Maps typically shows me a cycling time of 30 minutes when it take me 40 minutes and a walking time of 40 minutes when it take me 30 minutes. Either I walk much faster and cycle much lower than the typical person or Google Maps is not well calibrated to Singapore and India.

Things I Learned - 30 Nov 2025

Sun, 30 Nov 2025 00:00:00 +0000

This week, I learned:

Warp has a terminal agent feature - allowing Warp to control a terminal via text. I find that regular coding agents like Codex can do that too with tmux. For example, I opened a session and had Codex run commands in it while I watched. Here’s the guidance it needed:

# Create a new session
tmux new-session -d -s $SESSION 'uv run --with pandas,httpx,lxml python -iqu'
# Capture output to a log file
tmux pipe-pane -t $SESSION -o "cat >> /tmp/$LOG"
# Run a command
tmux send-keys -t $SESSION 'print(1 + 2)' C-m
# See output
cat /tmp/$LOG
# Capture the last 5 lines of the pane
tmux capture-pane -p -t $SESSION -S -5

Notes from Early science acceleration experiments with GPT-5 - via Claude
- LLMs are accelrating research because they are good at:
  - Literature search, especially across disciplinary boundaries
  - Generating and checking routine calculations
  - Proposing variations on known techniques
  - Identifying connections between disparate results
  - Producing first-draft code for well-specified problems
  - Explaining why certain approaches won’t work
- But they’re curently struggling with the following - though it’s a shrinking space
  - Genuinely novel conceptual leaps (but this is increasingly happening, e.g. Sawhney and Sellke’s problem #848)
  - Recognizing when it’s plagiarizing, e.g. when it “discovered” a proof for the Chevalley-Warning theorem which was copied from a Noga Alon paper - it wasn’t conscious of this
  - Knowing what it doesn’t know
  - Distinguishing important problems from unimportant ones
  - Understanding the “negative space” of mathematics (why certain problems are hard, why obvious approaches fail)
Anthropic introduced three excellent tool use practices that I expect will be adopted widely.
- Tool search: Don’t pass the tool definitions to the model. Model can ask for a tool search when needed
- Programmatic tool calling: Instead of calling a tool, it’ll return a Python program to execute that will call the tools! This is a huge win
- Tool use examples: Lets you specific examples of tool calls to guide th model better
- The Hacker News thread flags that CLIs solve these - but CLI updates are hard, while APIs auto-update.
With AI, some skills that beome more valuable are (and will soon be in short supply, hence need to be taught) are: #
- Problem formulation (“What question should we actually ask?”)
  - Traits: Curiosity (absolutely), systems thinking, comfort with ambiguity, metacognition (thinking about your thinking)
  - Practice reframing exercises (“What are 5 other ways to frame this?”), study great questions in your field, work backward from outcomes, learn adjacent domains. The “5 Whys” technique helps. Also: deliberately pause before diving into solutions—force yourself to spend time in the question space.
- Taste and judgment (“Is this response appropriate?”)
  - Traits: Pattern recognition from experience, cultural literacy, empathy, contextual awareness, aesthetic sense
  - How to strengthen: Immerse yourself in excellent examples, study spectacular failures (they’re more instructive!), get feedback on your calls, practice explaining why you made a judgment. Build a “swipe file” of great/terrible examples. The key is volume—you need lots of reps.
- Quality assessment (“Is this AI output correct?”)
  - Traits: Healthy skepticism, attention to detail, domain knowledge, logical reasoning, understanding of edge cases
  - How to strengthen: Study common AI failure modes, build verification checklists, practice the “does this make sense?” test, learn what “good” looks like in your domain, cross-reference claims. Develop your “bullshit detector” by analyzing why wrong answers feel wrong.
- Creative synthesis (“How do these ideas connect?”)
  - Traits: Associative thinking, wide knowledge base, playfulness, comfort with non-obvious connections, intellectual courage
  - How to strengthen: Consume diverse inputs outside your field, practice analogical thinking (“X is like Y because…”), use visual thinking tools like concept maps, study how innovations happen in other domains, give yourself permission to make weird connections. Read broadly—fiction, history, science.
- Domain expertise (“Does this solution work in reality?”)
  - Traits: Deep curiosity, persistence, willingness to get hands dirty, learning from failure, long-term commitment
  - How to strengthen: Deliberate practice on real problems, seek mentorship, study edge cases and failure modes, build things (don’t just read about them), learn your field’s history. The “10,000 hours” thing is real, but it’s quality hours that matter.
- Meta pattern:
  - Reflection loops: doing something, then analyzing why it worked/didn’t.
  - Exposure to excellence: you can’t develop taste without seeing great work.
Some more new CLI tools I installed:
- trash-cli: Alias rm to move files to trash instead of deleting permanently.
After a week of seeing ligatures in Fira Code, all other fonts look ugly. My favorite ligatures: !== ==> =» <–> (and every possible arrow) >= ||> ||- |- …
The first name, alphabetically (at least among Straive employees) is “Aabida” and the last is “Zyrene”. Something I would never have discovered working in a smaller company.
chokidar-cli is an easy way to run commands when files change, e.g. npx -y chokidar-cli '**/*.js' -c 'npm run build'
npx -y mapscii shows a map on the terminal. Not too useful, not maintained, but very interesting.
termsvg converts asciinema .cast files to animated SVG suitable for embedding in GitHub (e.g. via mise x github:MrMarble/termsvg -- termsvg export file.cast --minify). The animated SVG is ~10X larger than the .cast file. The GZipped size is fine but saving it as .svgz is not recognized by GitHub. In contrast, agg, the official asciinema-to-GIF converter, creates .GIF files that are only 5X larger. The most efficient seems to be embedding via asciinema.org

usql queries MySQL, Postgres, SQLite, MSSQL, Oracle, etc via a single interface. For example, usql 'mysql://rfamro:@mysql-rfam-public.ebi.ac.uk:4497/Rfam' -c "SELECT * FROM clan limit 3;". But DuckDB is more versatile, IMHO.

INSTALL mysql; LOAD mysql;
ATTACH 'host=mysql-rfam-public.ebi.ac.uk port=4497 user=rfamro database=Rfam' AS rfam (TYPE mysql);
SELECT * from rfam.Rfam.clan LIMIT 3;
SELECT * FROM 'file.xlsx' LIMIT 3;
SELECT * FROM 'file.csv' LIMIT 3;

Autistic and allistic people just have different communication styles. Autistic people have no trouble understanding other autists. They just happen to be in a minority which makes it seem like they have a social deficit. Conflict between Neurotypes
1 second = 10 tokens for OpenAI Realtime APIs. 1 second = 25 tokens for Gemini Live API
- 39 cents / hour on GPT Realtime Mini = 36 cents audio input + 3 cents text output
- 139 cents / hour on GPT Realtime = 115 cents audio input + 15 cents text output
- 30 cents / hour on Gemini 2.5 Flash Native Audio (Live API) = 27 cents audio input + 3 cents text output
Here are some AI experiments I’m planning to try with our marketing team:
- Video Generation: Create marketing videos from text scripts in minutes
- Poster Generation: AI designs high-conversion posters from brief text inputs - notably Nano Banana Pro
- Synthetic Persona A/B Testing: LLM agents simulate 100K+ user behaviors to test designs before real users
- LLM-Powered A/B Automation: AgentA/B system runs experiments with AI-simulated traffic
- Vibe Coding Landing Pages: Marketers build production-ready pages in hours vs weeks
- On-demand Landing Pages: Generate pages for automated campaigns/products without human intervention
- Brand Voice Cloning at Scale: Train on company content to ensure consistency across 1000s of pieces
- Persona-Driven Content Synthesis: Use 1B+ personas to generate diverse content perspectives
- Competitive Intelligence Briefing: Real-time monitoring across millions of data points + data storytelling
- Marketing Analytics with LLMs: AI agents analyze complex datasets for insights
- Brand Compliance Checks: Ensure all content meets brand guidelines automatically
- Autonomous Blog Squads: AI agents identify trending topics / internal content, create data stories ready for review
New skill unlocked: creating tutorials from talk proposals. I asked Claude to Write a Malcolm Gladwell article based on this talk description to teach me the topic and passed it this talk proposal: Your Causal Parrot might be lying to you. The story it wrote is very engaging and informative!
- LLMs “understand” causality because of training, but lack a world model to extrapolate to new situations.
- Giving them tools to reason (e.g. causal models, sub-agents to explore root causes) will help.
A cool Gemini 3 Pro hack: convert satellite imagery into stylized maps! Bilawal Sidhu
Running sub-agents in tmux helps avoid timeout cancellation, and hence allowing resuming Peter Steinberger

Things I Learned - 29 Jun 2025

Sun, 29 Jun 2025 00:00:00 +0000

This week, I learned:

“People are great at feedback on what you are doing wrong. They are not so good at telling you how to fix it. They don’t know you that well.” Amit Kapoor
Perfect Cursors makes periodic cursor positions animate smoothly by interpolating on a spline**
CloudFlare and Vercel now support sandboxes where you can execute code. The price is not so low that we can execute for free in bulk but works well infrequent or batched code execution. Simon Willison
Here’s how I’m using ffmpeg for video recording & editing.
- To record screen at 5 frames per second, I run an abbreviation screenrecord which maps to:
Gemini CLI has a generous free tier and uses Bootstrap over Tailwind Ref #ai-coding
Cloudflare has a native agents SDK that looks good, especially for CloudFlare users. Ref
There are several brands with recognizable chart style guides. It’s possible to generate style guides for these from the charts, but applying them via matplotlib is almost #impossible today. ChatGPT
Hyperfine is like %timeit for the shell. Written in Rust
⭐ Vertical AI is a moat against AGI. Specialization reduces hallucinations. Custom workflows and regulations are sticky and defensible. We need to start selling to users, not IT, though. Ref
When AI automates a task, the bottleneck shifts. AI process re-design is about reworking the process around the new bottleneck, and iterating quickly.
- With coding, it’s testing, reviewing, deploying, use-case identification.
uvx git-smart-squash re-organizes haphazard commits using LLMs. git-smart-squash #ai-coding
GitHub offers a free Docker container registry. Simon Willison
There are three major areas where humans either are, or will soon be, more necessary than ever: trust, integration and taste – NYT. Anil. To deal with this:
- Learn things that might grow in importance, like:
  - Data modeling
  - APIs
  - Code reviews
  - Drawing and 3D modeling
  - Narrative storytelling
  - Design
  - Movie making
  - Statistics
  - Sceptical fact checking
  - Continuous AI auditing e.g. awesome-continous-ai or automated-auditing
  - Zero knowledge proofs
  - Homomorphic encryption
  - Privacy-preserving computation
  - Fingerprinting and watermarking
  - Governance frameworks
  - Ethics and AI dilemmas
  - Negotiation
  - Change management
  - Remote working, management, hiring
  - Creating attention scarcity
  - Local cultures
- Work with people of growing importance
  - People designing products in regulated industries
  - Cross domain experts
  - Art developers, game makers, designers
  - System thinkers. Economists, ecologists, system planners. People who look for second order effects.
- Live in cities that might play a bigger role in the future
  - Cities like Singapore and learn how it builds civics trust, creates digital IDs.
  - Cities like Bangalore and Hyderabad and learn how they grow tech talent
  - Creative cities like Paris, Seoul, Mexico City, Berlin, etc. on sabbaticals to taste hubs
- Try to:
  - Build auditing credentials and IP
  - Audit your calendar for what AI can do. Have it interview you
  - Practice sceptical fact checking and audit
A clever way to test a library’s quality is to have LLMs write code from docs and test it. Failing libraries have flawed code/docs. Improve. Ref #ai-coding
Common Pile is an 8TB open dataset for LLM training that includes ArXiv, PubMed, StackExchange, GitHub, IRC, Regulations.gov, Patents, UK parliament, books. Easier than scraping.
A useful way to have reasoning models do deep-research-like work is to have them “First, create a plan to solve the problem, clearly listing the objective, approach, and output. Then follow the plan.”
DE-COP is a method to check if LLMs were trained on private content. GPT-4o was trained on O’Reilly books, based on this method. Ref
LLMs are more persuasive than humans. But repeated exposure reduces the effect. Ref
Phoenix.new uses live views to publish apps as it codes. The testing framework looks at the screen while it codes and fixes errors. It commits every change
Anthropic system prompt asking Claude to pursue its goals led to self preservation behavior. Ref
The hungrier I am the better the food tastes. A good reason to eat less quantity and frequency
You can purge the jsDelivr cache manually. Helps if you released a new version of a package and way to purge an alias (e.g. https://cdn.jsdelivr.net/npm/your-package@1)
XConvert is a convenient online app to compress .webm videos. Not great design but fairly good compression.
You can draw a treemap of import times via python -X importtime app.py > timing.txt and then paste them at https://kmichel.github.io/python-importtime-graph/.
PyOpenLayers adds interactive mapping via OpenLayers to Marimo and Jupyter.
In a TechCrunch interview with Jared Kaplan has was asked if Anthropic is becoming less safety conscious because they released Opus 4 which blackmails. Kaplan replied that they have stronger testing and higher transparency, so they’re more likely to share AI dangers early. Great positioning! Conversations are about perspective change and this nailed it.
The system prompts for Anthropic misalignment evals are a fascinating read.
AI PR Watcher tracks GitHub pull requests from Codex and other LLMs. Codex is way ahead of anything else on volume and success rate. Devin is next on volume, Cursor is next on success rate.

Things I Learned - 11 May 2025

Sun, 11 May 2025 00:00:00 +0000

This week, I learned:

snapdom is a fast, light, element capture alternative to html2canvas but doesn’t work well with non-CORS images or iframes.
Sli.dev is a Markdown slide language. Similar to Marp
Don’t split your code into microservices until you need to scale. Ref
Vibe coding is like getting others’ code to work, which is exactly what most devs do. Simon Willison #ai-coding
Tofu Yakitori is a Japanese dish. It’s like a dhokla. Marinated tofu cubes brushed with that sweet‑savory tare (soy, mirin, sake, a hint of sugar), then grilled until caramel‑charred. One of the better (tasty + different) dishes I’ve had recently. I used ChatGPT to remind me of the dish name.
Trust, attitudes and use of artificial intelligence surveyed ~1,000 people across 47 countries on their views on AI. PDF
- Emerging economies trust and use AI more. It’s an opportunity to leapfrog.
- 26% of students use AI daily (vs 17% employees). Efficiency is the main benefit.
Gemini APIs now have automatic caching for 75% cost reduction if message is >1K (Flash) or >2K (Pro) tokens. Ref
YOLO is much better than Gemini at object detection. Use for pro-processing. Ref
Using [[n]] is probably the best citation format for inline search references in RAG. ChatGPT
⭐ Double-checking is surprisingly efficient since LLM hallucinations are mostly uncorrelated. LLMs perform human tasks (e.g. classifying customer support messages) at ~85% accuracy. This might be unacceptable. But by asking 2 moderately correlated LLMs and double-checking discrepancies, we reduce automation by ~20% but reduce errors to 0.25%. Triple-checking reduces automation by ~25% but errors to under ~0.01%! Ref
Anthropic introduces web search in the API at $10 / 1K searches. Here’s how it compares:
- $0.1: DuckDuckGo Search API (RapidAPI) (monthly pricing)
- $3: Brave Search API
- $5: Google Custom Search JSON API
- $15: SerpAPI
- $10: Zenserp
- $10: Anthropic Web Search Tool
- $25: Bing Search API
- $35: Gemini API
- $35: OpenAI API
India attacked Pakistan!
⭐ When writing notes, summarize at the end of the day the learnings and next steps.
GitHub does not let you control the cache duration, but there are many creative workarounds. ChatGPT
- HTML meta tags: <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate">
- Use a service worker (blog)
- Proxy through a CDN. Cloudflare, Netlify
- Move to another static host: S3 + CloudFront, Heroku, Vercel, Surge, Firebase Hosting
Notes from the PromptEvals paper:
- Good evals must be:
  - Objectively MEASURABLE (even if by an LLM). Otherwise, we won’t know if it’s right.
  - Directly RELEVANT to the input/prompt. Otherwise, we’re not evaluating the input.
- Typical evals fall into 6 categories
  - Structured output: Adhere to a schema (Markdown, HTML, DSL, JSON + Schema)
  - Multiple choice
  - Length constraints: N characters, words, sentences, list items, etc.
  - Semantic constraints: Exclude terms, topic relevance, follow grammar, etc.
  - Stylistic constraints: Style, tone, persona
  - Prevent hallucinations: Factual accuracy. Instruction following

Things I Learned - 22 Dec 2024

Sun, 22 Dec 2024 00:00:00 +0000

This week, I learned:

What to use for hosting: ChatGPT
- GitHub Pages: Static websites, medium files
- Cloudflare Pages: Static websites, global delivery
- Vercel: Frontend frameworks (e.g. Next.js) with high DX and ISR, small files
- Netlify: JAMstack projects, minimal back-end, moderate files
- Glitch: Small static projects
- Render: Full-stack apps requiring databases and server-side compute
- Firebase Hosting: Small sites, limited large files
- Archive.org: Public archival, large files
- Google Drive: File sharing, large files
- Dropbox: File sharing, moderate files
- Cloudflare R2: Static assets, large file delivery
Anthropic defines agents. Building effective agents + Cookbook
- Augmented LLMs are LLMs enhanced with augmentations such as retrieval, tools, and memory.
- Workflows are systems where LLMs and tools are orchestrated through predefined code paths.
  - Prompt chaining: Pipe each LLM output to the next LLM. A->B->C->Z. E.g. Write report, then translate. Extract results, then verify them. Successively ask follow-up questions.
  - Routing: One LLMs decides which other LLM to call next. A->B|C|D->Z. E.g. Evaluate complexity, then pick the right model. Classify request time, then pick the right prompt.
  - Parallelize: Sectioning (and Orchestrator-workers): Break tasks into independent subtasks, then aggregate. A->B+C+D->Z. E.g. Evaluate contracts against different clauses in parallel.
  - Parallelize: Voting: Run same task multiple times, then vote. A->B+B+B->Z. E.g. Review code for prompt injection using different prompts. Evaluate content safety with different thresholds.
  - Evaluator-optimizer: One model checks another in a loop. A->B->A->B->…->Z. E.g. Literary translation. Self-healing code. Policy violation checks.
  - Human-in-the-loop Checkpoints: The workflow explicitly requests human review at certain stages. A->B->(Human)->C->Z. E.g. Sensitive content review. High-stakes decision making. Ambiguous tasks.
- Agents are LLMs that dynamically direct their own processes and tool usage, consulting tools or the user as needed.
To download YouTube subtitles, use: yt-dlp -q --skip-download --convert-subs srt --write-sub --sub-langs "en" --write-auto-sub --print "requested_subtitles.en.url" "$url" Simon Willison
o1-preview diagnoses better than doctors. Harvard
OpenAI’s release of ephemeral tokens via sessions (valid for 1 minute) are a useful way of exposing apps for public demos. Currently it works only for the Realtime API, though.
SpreadsheetLLM is a way of encoding spreadsheets in an LLM friendly format. It’s good for 1K+ rows. For lower, Markdown > XML > HTML. However, Table Meets LLM suggests that HTML > XML > Markdown, so this is unclear.
#HARD prompt. Ask video generators like SORA to generate text in videos. It is of average quality.
GPT 4o Mini Realtime was released. A realtime conversation will cost ~50c/hr. About 36c for input, 72c for output. (I extrapolated from the 6c/min audio input cost for GPT 4o Realtime when it was $100/MTok. GPT 4o Mini Realtime is $10/MTok input and $20/MTok output.)
This is an interesting way to understand software. Generate a Mermaid sequence diagram showing interactions based on this code. Ref
The King James Bible and all Harry Potters, each, are about $1M tokens (rounded off).
markdown2 is the new de facto Markdown library for Python.
Claude 3.5 Sonnet is way ahead of competition on the LMSYS Webdev Arena
Raspberry Pi 5 has a faster CPU, more RAM and GPU, 4K support, multiple USB 3 ports
Government websites like the official press releases cannot be crawled from outside India. Hence the need for server farms in India!

Things I Learned - 01 Dec 2024

Sun, 01 Dec 2024 00:00:00 +0000

This week, I learned:

Gists are a good place to store static files for posterity as well as throwaway files. But, they’re just git repositories. So there may be no advantage over GitHub repos.
GPT-4o Audio supports tone control via XML tags like <cough>..., <laugh>..., etc. But at ~$15/hr of output, it’s too expensive. Ref
Mridula’s son gave a live commentary of what he was doing on Minecraft and ChatGPT gave him live evaluation and coaching. E.g. “Great strategy! Getting to the launch pad early can give you a huge mobility advantage. Making the bridge wider is also a smart move to prevent accidental falls. With this plan, you’re setting yourself up for success. This is a great way to interact with LLMs.
Gemini’s JSON mode returns JSON with keys in alphabetical order. I think. Emperical evidence. This is unlike OpenAI which explicitly returns the keys in the order specified.
- To solve this, order the keys alphabetically.
HTMX focuses on HTML over JS. Like server responses being HTML snippets not JSON. But I need front-end over back-end. Client side apps. HTMX doesn’t help much there, e.g. templating, or just plain JS code.
- htmx client side templates do can convert JSON to HTML.
I installed the OpenAI Desktop App as well as Claude for Desktop. They take up too much RAM (260MB and 750 MB respectively on startup - though this varies.) The ChatGPT web page takes ~100MB incrementally, so I wrote an AutoHotkey script to switch to the first open (or recently closed) ChatGPT tab on Brave.
I tried LIDA from Microsoft, after almost a year of its release. A few notes:
- Just running uvx lida ui --port 8080 --docs works.
- But I needed to use export TCL_LIBRARY=C:/Users/Anand/AppData/Roaming/uv/python/cpython-3.13.0-windows-x86_64-none/tcl/tcl8.6 to point it to my TCL installation for charts to work. I also chose to export OPENAI_BASE_URL=https://llmfoundry.straive.com/openai/v1
- I also chose to replace gpt-3.5-turbo-0301 (the default model) with gpt-4o-mini in lida/web/ui/component*
- It’s quite impressive.
OpenAI allows multiple system messages. I learned this browsing through the LIDA prompts.
Anthropic’s Model Context Protocol lets any apps integrate with LLM Apps. LLM Apps are becoming the new operating system. Competitors, beware.
I spoke at Automating Data Visualizations using LLMs at SUTD. Apparently, using LLMs to write code is much more common than writing code to use LLMs. I ran a quick quiz.
- Have you used ChatGPT or any LLM? 35 / 35 raised their hands.
- Have you written code using an LLM? 34 / 35 raised their hands. (I was impressed.)
- Have you uploaded a spreadsheet to an LLM for analysis? 15 / 35 raised their hands.
- Have you programmatically called an LLM API? 6 / 35 raised their hands.
With LLMs, fostering innovation is a new path to profitability. Companies are increasing innovation team sizes. Productionizing that is the next. Some initiatives are:
- Convert popular demos into starter kits
- Create and evangelize trainings on solutions and solution techniques
- Create larger pools of capacity to build innovation and productionize it
Andrew Ng Explores The Rise Of Al Agents And Agentic Reasoning | BUILD 2024 Keynote
- Innovation is now a path to production. People are able to build 20 prototypes at the cost of one and see which sticks
- Machine learning is much faster. Things that took months now date days. But engineering and evaluations are only slightly faster and have become a bottleneck
- A good analogy to zero shot prompting is to ask a person to write an entire essay without pressing backspace even once
- Andrew scenes to align with the line chain definition of agentic workflow, which is about agents being able to craft their own control flows
- People find it very easy to understand agentic workflows once they read through the code
- Reflection or feedback is a useful agentic pattern
- In multi-agent collaboration, it may be the same underlying model that is acting as different agents. But just like we find it useful for the same CPU to run multiple processes and each application is its own abstraction, agents of useful abstraction
It’s hard to summarize a large document using RAG. But you can directly add answers to such questions into the corpus, e.g. by adding a “summary” section, and other answers to common questions.
CloudFlare workers can bundle any kind of files, including text, data, and WASM. Docs
AssemblyScript can compile TypeScript to WASM. Here’s what I learnt
Here’s a convenient pattern to git commit a directory but nothing else in it (e.g. a build/ directory). Add a .gitignore file with * followed by !.gitignore. Only the .gitignore file is tracked.
Ultravox lets you build voice agents at 5c/min = $3/hr (OpenAI is 6c input, 24c output). Or clone their repo.
- Idle call time is counted towards cost. So cost may be higher than OpenAI.
- Voice cloning quality is average. Very distinctive voices are just partly identifiable.
- Supports tool calls (from their server).
- Their API is simple but the docs have minor errors (e.g. a trailing comma in the JSON, which leads to an error) reducing confidence.
LLMs may be good at derived data generation. For example, given a database schema, what derived columns would be useful? What derived views would be useful?
The O1 model does not have a mechanism to control the amount of tokens to spend on reasoning. DeepSeek R1 might, but the API is not out yet.
The OpenAI Desktop App can interact with native applications, e.g. read from Terminal, VS Code, etc. This takes it on a path to becoming a copilot for ANY apps. Putting every copilot app and every LLM integration under threat.
Crawl4AI and Firecrawl are tools / libraries to convert websites into LLM Friendly Markdown and extract structured data using LLMs.
Don’t try and solve specific problems. Pass the entire context to an LLM and get a comprehensive solution. Most doctors, for example, ask specific search-like questions instead of uploading the entire case history and asking for a diagnosis, and perform workse than LLMs. Ethan Mollick

Things I Learned - 17 Nov 2024

Sun, 17 Nov 2024 00:00:00 +0000

This week, I learned:

Anthropic has single-plage docs for LLMs. Condensed version and Full version
Malcolm Gladwell on the importance of self-correction
- Belonging to multiple social worlds is a good way to defend against no longer being good at what you used to be. Diverse values and social groups help.
- Self handicapping explains a lot about the world. You study late for a maths test - so you can fail for lack of trying, not aptitude. Ecosystems (e.g. sports teams) mitigate self-handicapping.
- You don’t have to be good in athletics to get the benefits. A slow runner gets the same discipline, pumping up, etc that a fast runner does
- Mono cultures are good to accomplish a known mission. Diversity is good to pivot during uncertainty. So, localize mono cultures
- Diversity helps only if there are sufficient numbers, or if they have enough power to change the organization’s thinking.
Use a standardized password strategy, e.g. use the month like GramNov2024 (via Namit)
Gemini has an OpenAI compatible API. Gemini Docs
Ethan Mollick says Claude is solving MBA case studies well. x.com
LLMs pay a lot of attention to the first 6 tokens. Ref
This is an interesting article on “UI in the age of Gen AI”. Ref
Google Open sourced Alphafold 3. Repo
Cloudflare R2 has the same API as S3 but is cheaper
Prefect.io is a good alternative to Airflow / cron. Can use for synchronisation tasks, e.g. Drive to server. But no Auth, UI params or config.
Gemini transcription does not give accurate timestamps. Whisper does. But the quality of transcription is similar.
Pass a complex data structure to Claude.ai and have it create an app to visualize it. It does well. Simin Willison
Tech Council Ventures and Sunicon VC invest in early stage startups, and aloso provide them technology support (via Naveen)

Fri, 20 Sep 2024 04:07:16 +0000

Looks like XML tags are the best way to structure prompts and separate sections for an #LLM. It’s the only format that all of Anthropic, Google, and OpenAI LLMs encourage.

For example:

… … … …

Anthropic Docs: https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags OpenAI Docs: https://platform.openai.com/docs/guides/prompt-engineering/strategy-write-clear-instructions Google Docs: https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/structure-prompts

Alternatives are using JSON, Markdown, templating formats like Mustache/Jinja, etc.

Even Llama’s system tokens seem a little XML-like. https://github.com/meta-llama/llama3/blob/main/llama/tokenizer.py#L61-L74

Personally, I’ve been using Markdown so far. But it’s time to switch over. (Only on the prompt side. On the generation side, Markdown still seems the best.)

How fast are LLMs in production?

Sun, 01 Sep 2024 05:29:54 +0000

At Straive, we use an LLM Router. Since ChatGPT, etc. are blocked for most people, this is the main way to access LLMs.

One thing we measure is the speed of models, i.e. output tokens per second. Fast models deliver a much smoother experience for users.

This is a different methodology than ArtificialAnalysis.ai. I’m not looking purely at the generation time but the total time (including making the connection and the initial wait time) for all successful requests. So, if the provider is having a slow day or is slowing down responses, these numbers will be different.

Hopefully this gives you a realistic sense of speed in a production environment.

Here’s the speed of models with at least 500 requests over the last 2 weeks. I’ve grouped the models based on speed grades

Grade 1: 100+ Tokens / second. Groq is clearly serving the Llama 3 models at blazing speed. No surprises there – except why Groq still doesn’t let me pay. The free tier is open with generous rate limits and the Pay per Token model has been “Coming Soon” for several months now (and I’ve no complaints 🙂).

Grade 2: 70+ Tokens / second. Anthropic’s Claude 3 Haiku is the next fastest class of models, but Claude 3.5 Sonnet is surprisingly fast, almost as fast as Haiku and over 70 tokens per second. This is impressive.

Grade 3: 50-60 Tokens / second. OpenAI’s GPT 4o models are almost as fast. It’s interesting that GPT 4o and GPT 4o mini are at about the same speed! GPT 3.5 Turbo is not far behind either. Perhaps OpenAI increases capacity for slower models?

Grade 4: 30-50 Tokens / second. Gemini 1.5 Flash is a much, much slower than the benchmarks - maybe we’re doing something wrong. Azure’s GPT 4o service is about twice as slow as OpenAI’s, and comparable is speed with Gemini 1.5 Pro.

Grade 5: <20 Tokens / second. Azure’s GPT 3.5 Turbo and Google’s Claude 3 Sonnet are among the slowest ones. These are older models on third-party infrastructure, so I suspect they’ve been given weaker infrastructure (unlike OpenAI which is serving GPT 3.5 Turbo at 3X the speed Azure does.)

Drivers of speed

Here’s what I’m taking away (informally):

GPU architecture is the biggest driver of speed. Groq is FAST! Hopefully, the fact that they won’t let us pay isn’t a red flag that the service will vanish.
How companies operate seems the next biggest driver. Anthropic’s models are consistently faster than OpenAI’s which are faster than Google’s.
Companies run their own models faster than cloud providers. OpenAI is faster than Azure, and Anthropic is faster than Google for the same models.

Things I Learned - 25 Aug 2024

Sun, 25 Aug 2024 00:00:00 +0000

This week, I learned:

Karya.in is creating high quality datasets. Suhel mentioned them
An 8-year old uses Cursor.ai to code
Hermes 3 has special tokens like <SCRATCHPAD>, <RESTATEMENT>, <THOUGHT_*>, <PYDANTIC_SCHEMAS>, <SCHEMA_*>, <REASONING>, <INNER_MONOLOGUE>, <PLAN>, <EXECUTION>, <REFLECTION>, <THINKING>, <SOLUTION>, <EXPLANATION>, <UNIT_TEST>, etc. This extends the capability dramatically.
Lumentis creates docs from transcripts and text
LLMs write worse code in JSON than Markdown
Copilot’s system prompt calls a search_enterprise(query: str) tool and a hint(M365Copilot_language: str) tool as assistants.
Anthropic Prompt Caching is 90% cheaper to use and 25% costlier to create. So if there’s a 27% chance it’ll be re-used, cache it.