fine-tuning on S Anand

Things I Learned - 09 Nov 2025

Sun, 09 Nov 2025 00:00:00 +0000

This week, I learned:

“But when an identity based belief was challenged, the brain responded as if under physical attack.” Why Engineers Can’t Be Rational About Programming Languages
Notes from How to build a cult, Lulu Cheng, The Knowledge Project podcast
- Conviction is infectious.
- Communicate at the INTERSECTION of interests. Learn theirs
- Begin with “why your story matters to them” (first sentence). That beats “how you tell it” > “where you tell it”.
- The easiest way to align with an audience is to find your community.
- Humor, curiosity, awe, any strong emotion is a hook.
- Culture has momentum. Best way to break it is to show an alternative that works. People will copy that
- REPEAT messages over and over with complete CONVICTION to convince people who TRUST you. That works, but you need all three.
- Trust builds from likeability, repeated exposure, common beliefs.
- An excellent way to defend against online criticism (when it matters) is to just SHOW UP and THANK them for feedback.
- Serious reputational damage must either be fixed immediately - or you live with it forever.
- Between a story and statistics, the story will always wins. Never fight a story with a statistic. Dig into your statistics and uncover BETTER stories.
- ⭐ Prebuttals are a great idea. Start with all possible criticisms yourself and diffuse them. The other person has nothing left to say
- Sparring keeps you sharp. Spar with LLMs.
- To defend, show how the attack targets other people, increasing the surface area. Show how the SPECIFIC attack targets a larger group. Create a SPECIFIC cause worth fighting for.
- Each role has specific objective to optimise for. The leader’s role is to balance across these.
- Cheerleader effect. People look beautiful next to a cheerleader. Associations taint.
- Each person has dozens of aspects to their persona. We cannot remember all of them. Each person can make a choice on who they project themselves to be in any group. Shaping their persona.
~~The Rainbow CSV extension may be causing delays (infinite spinner) when pasting Markdown in VS Code. Restarting it seems to fix the issue.~~
⭐ Claude scientific skills is a collection of skills teaching Claude how to use scientific libraries, databases, and APIs across several domains. This may be a good example of a non-trivial skill library - that is hard for AI coding agents to infer by themselves.
Notes from How I use every Claude Code feature
- Use AGENTS.md as guardrails, not a manual. Document what it gets wrong.
- Use self-documenting tools/APIs rather than documenting.
- Docs: Explain why and when to read each doc.
- Never say “Never.” Explain when to which which alternative.
- Prefer CLIs for stateless tools, MCPs for stateful, authenticated, or complex (e.g. Playwright).
Coding agents work well with version control. Simon Willison
- Break up uncommitted changes into small commits
- Rewrite branch history for readability
- Use gh CLI to fetch line-wise comments from a PR and make requested changes (e.g. renaming, refactoring, adding types, etc.)
⭐ When using MCPs or tools with private data, “color untrusted content in red, unsafe actions in blue, and never mix colors.” Good advice.
⭐ DeepWiki offers a codemaps feature that explains code in an interactive way. It shows a structured explanation on the left. You can click on any note to see the code on the right. It’s an effective way to understand how a library or tool executes a task. Here’s an example of how Mermaid works.
Gemini offers RAG with free storage. RAG costs are quite high. This simplifies the process a lot. But I tried running the sample program and after an hour, it still had not completed uploading a single file. Best to wait and watch.
OpenRouter supports embedding models using an OpenAI-like API
Kimi K2 Thinking seems popular because
- It’s an open-weights model on par with the top models on Humanity’s Last Exam (text-only) and BrowseComp
- Can run 200-300 tool calls without human guidance
- 4x cheaper than GPT-5 with low tokens (32B active on 1T parameters, INT4 quantized)
Based on responses to Simon Willison’s question, ChatGPT
- Fine-tuning helps when:
  - Lower latency, e.g. for type-ahead, at lower cost (37 mentions)
  - Structured extraction, parsing and classifiers, e.g. postal address, detecting secrets (18 mentions)
  - Custom vision models, e.g. check containers (12 mentions)
  - Domain-specific code and stacks (niche languages, stack-specific generation, text→SQL) (11 mentions)
  - … and a long tail.
- Fine tuning does not help:
  - When A base model plus prompting or RAG does as well or better (15 mentions)
  - When you risk being leapfrogged by a new release (4 mentions)
  - When cost and data do not justify the ROI (3 mentions)
The data I can export from my Android phone includes the below. 🟢 indicates it’s tracked. 🟡 might need action, e.g. enabling / coding. #
- 🟢 GPS/GNSS location (current & history). Turn on device Location. If you want a timeline you can export, enable Google Location History and later export via Google Takeout → Location History (JSON/KML).
- 🟡 GNSS raw measurements (engineering traces). Android exposes GNSS “raw” logs on many devices; capture with dev tools or logging apps if supported (intended for research). See GNSS Raw Measurements API.
- 🟢 Wi-Fi scans (nearby SSIDs/BSSIDs). Toggle Location scanning → Wi-Fi scanning in Location settings; apps need location permission to read results.
- 🟡 Wi-Fi RTT distance to APs (indoor ranging). Apps can use Wi-Fi RTT (802.11mc/az) to measure distance to compatible APs; requires location permission.
- 🟢 Bluetooth proximity/traffic. For packet-level logs, enable Developer options → Enable Bluetooth HCI snoop log, then pull /sdcard/btsnoop_hci.log (Wireshark).
- 🟢 Cell towers (IDs, signal strength). Apps can read via TelephonyManager (e.g., getAllCellInfo()), with appropriate telephony permissions.
- 🟢 Activity recognition (walking, running, in vehicle). Apps must request ACTIVITY_RECOGNITION (runtime) from Android 10+.
- 🟢 Steps (step counter / detector). Use sensors API; from Android 10+ you must declare ACTIVITY_RECOGNITION to access step counter/step detector.
- 🟢 Accelerometer / gyroscope / magnetometer streams. Apps read via SensorManager; some high-rate reads require HIGH_SAMPLING_RATE_SENSORS.
- 🟢 Ambient light / proximity. Read via SensorManager; typically no special permission.
- 🟢 Google Fit data (steps, workouts, heart rate from wearables, etc.). Manage and export from Google Fit / Google account Download your data.
- 🟢 Contacts. MIUI → Settings → System apps → Contacts → Import/Export to .vcf (vCard).
- 🟢 Call history / SMS (device). MIUI local/cloud backup can include call logs & messages; export by creating a local/Cloud backup and downloading. Note: 3P apps can’t read call/SMS logs unless they’re the default dialer/SMS.
- 🟡 Gmail, Calendar, Contacts (Google). Export via Google Takeout (MBOX/ICS/CSV etc.).
- 🟡 WhatsApp / Telegram / Signal chats. Use in-app exports: WhatsApp → Export chat, Telegram Desktop → Export, Signal → encrypted backup.
- 🟢 Advertising ID. View/reset in Settings → Google → Ads (wording varies), per Google help on Ad ID reset.
- 🟡 Per-app screen time / unlocks / opens. Third-party “usage” apps (e.g., analytics or “digital wellbeing” clones) require Usage Access (PACKAGE_USAGE_STATS). Use Android’s UsageStatsManager or apps that export CSV. Stock Digital Wellbeing does not offer an export.
- 🟡 Notification history (last 24h). Settings → Notifications → Notification history → On. OEM-optional, but present on most devices. Viewable once enabled.
- 🟡 Notification content stream (live). Grant an app Notification access to capture/export notifications going forward. (User-granted API via NotificationListenerService.) |
- 🟢 Per-app data usage (mobile/Wi-Fi). Apps/ADB can query NetworkStatsManager; Settings shows per-app totals. Advanced dumps via adb shell dumpsys netstats.
- 🟡 Wi-Fi detailed logs. Developer options → Enable Wi-Fi verbose logging for richer diagnostics.
- 🟡 Bluetooth packet logs. Developer options → Enable Bluetooth HCI snoop log; export file and analyze in Wireshark.
- 🟢 Per-app storage usage. Apps/ADB can query StorageStatsManager; Settings shows per-app storage.
- 🟡 Photo/video metadata (EXIF incl. location). Enable “Save location” in Camera app to embed GPS in EXIF; export files normally (EXIF remains). |
- 🟢 Downloads & file metadata. Use a file manager or connect via USB; metadata is in the files themselves. |
- 🟢 Battery usage history (per-UID/app), wakelocks, jobs. Generate adb bugreport and analyze with Battery Historian or dumpsys batterystats.
- 🟡 System/device logs (logcat). You can view via ADB/Android Studio. Android restricts 3rd-party access to system-wide logs for privacy.
- 🟢 Developer quick tiles (Sensors off). Developer options → Quick settings developer tiles → Sensors off to globally cut Camera/Mic & SensorManager sensors on demand.
- 🟡 Google Takeout: one-stop export for Location History (Timeline), Gmail (MBOX), Calendar (ICS), Google Photos, Drive, YouTube, Fit, etc.
MacroDroid, Automate and Tasker sound like powerful Android workflow automation tools. Some uses I can put it to:
- Automatically upload recordings to Dropbox
- Turn off hotspot when I reach office
- Vibrate if I’m walking slowly
Adding <link rel="alternate" type="text/markdown" title="LLM-friendly version" href="/llms.txt"> is an emerging approach for pointing to LLMs.txt. It works. I asked Codex to read the CloudFlare vitest page. It read the file truncating the middle, found the <link rel="alternate" type="text/markdown" href="https://developers.cloudflare.com/workers/testing/vitest-integration/write-your-first-test/index.md"/ link in it, and reasoned “Considering fetching markdown instructions” and fetched the Markdown page. Giles’ Blog
toon is a YAML-like format that’s LLM friendly and especially token-efficient (CSV-like) for tables. You can convert back and forth between JSON and toon.
Food printing applies 3D printing techniques to create real food items. Given the art that this can create, I expect at least some adoption in niche restaurants.
PMTiles lets you store map tiles as a single-file archive that libraries like MapLibre can read. Useful to avoid tile servers.
Mirrow is a CLI SVG animation builder that converts a DSL to animated SVGs. However, it may be easier to use an LLM to create the animated SVG directly with SMIL than learning Mirrow (or teaching the LLM Mirrow).
⭐ One approach to giving memory (“episodic memory”) to coding agents is to allow them to search their logs.This gives them access to past discussions about a repo or other repos.
To configure Gemini CLI with an AI router, set:
- "security.auth.selectedType": "gemini-api-key" in ~/.gemini/settings.json
- export GOOGLE_GEMINI_BASE_URL=https://llmfoundry.straive.com/gemini/ (or your AI router base URL for Gemini)
- export GEMINI_API_KEY=... (your AI router API key)
Passing a HAR export to an LLM to build a scraper is a powerful idea! Lessons from Diagram Chasing
Addy Osmani’s Gemini CLI tips are practical guides to using any coding agent, not just Gemini. I learnt about:
- Run shell commands with !, e.g. !ls -la or even !bash. It’s added to the chat.
- On-the-fly tool creation: ask it to write code for the task on the fly.
- Use it for system optimization, e.g. editing dotfiles, system customization, log error analysis, etc.
- Run GEMINI_SYSTEM_MD=... gemini -p "task" --yolo --format json < input.txt to run Gemini with a different system prompt and feed it input.txt to run in a pipeline. (FYI: Codex does not send a default system prompt, so there’s nothing to override.)
- There is a Gemini CLI Show and Tell thread with examples. This include Janitor AI, a Gemini CLI session viewer, etc.
- Hands on with Gemini CLI has several Use cases to try out. Renaming photos and organizing files are clever ones.
- AGENTS.md can be used like a decision log - rules, styles, or preferences that evolve over time - on a per-repo basis. Gemini’s /memory add feature helps with this.
- gemini --checkpointing is a useful “undo” feature. /restore rolls you back to a specific checkpoint. The overhead is small.
- Caching is only available with API key or Vertex AI, not OAuth login as of now
OpenAI TTS costs are confusing. But in short
- TTS-1 costs $15 / MChars (max 4,096 chars per request), which ends up at ~86c / hour
- GPT-4o Mini TTS costs ~$16 / MChars (max 2K tokens which is ~7,000 chars per request), which ends up at ~88c / hour. Very similar cost, effectively
- TTS-1 HD is twice TTS-1.
OpenAI has a usage API that provides cost as well as usage for completions, images, audio speeches, etc.
- These require an organization admin key
- Cost API: curl "https://api.openai.com/v1/organization/costs?start_time=$TIMESTAMP&project_ids=$PROJECT_ID&group_by=line_item"
- Audio speech usage API: curl "https://api.openai.com/v1/organization/usage/audio_speeches?start_time=$TIMESTAMP&project_ids=$PROJECT_ID&group_by=model"

Things I Learned - 10 Nov 2024

Sun, 10 Nov 2024 00:00:00 +0000

This week, I learned:

OpenFreeMap is a free embeddable OpenStreetMap tile server. You can use MapLibre GL (more features) or Leaflet (simpler) to render it. It offers styling and self-hosting.
Zapier Actions are an easy way to set up custom actions like GMail / Google Calendar APIs for GPTs, since GPTs’ callback URLs keep changing. But they fail often, and don’t work on mobile. At least for me.
LLM Vision Use Cases in manufacturing and earth sciences (via Shivku)
- Automated geoscience image descriptions Ref
- Interpret Wind Turbine photos and charts, construction monitoring, equipment maintenance & charts Ref
- Forecast weather based on cloud photos! Ref
- Analyze thermal image of solar panels, electroluminescence images for warranty claims, ROI estimates from Google Sunroof rooftop images Ref
- Corrosion detection in electricity towers, turbines, storage tanks, penstock. Interpret non-destructive test images Ref
Google counts auto-completion when saying “25% of all the code is written by AI at Google”. “It’s a helpful productivity tool but it’s not doing any engineering at all. It’s probably about as good, maybe slightly worse, than Copilot.” YCombinator
Workflow for AI video creation: Use Meshcapade (meshcapade.com) to generate body movement of a 3D-rendered character. Pass that video to Runway’s video-to-video model to generate any visual. Add music from Suno Ref
Someone sorted the X and Y columns independently for regression. Ref
Android keyboard learning only sends model changes back to server and not local keywords. Model changes are aggregated! Ref
Here is a prompt for audio transcription using Gemini. Ref
- Transcription: Accurately transcribe the audio clip in the original language. Include all spoken words, fillers, slang, colloquialisms, and any code-switching instances. Pay attention to dialects and regional variations common among immigrant communities. Do your best to capture the speech accurately, and flag any unintelligible portions with [inaudible].
- Translation: Translate the transcription into English. Preserve the original meaning, context, idiomatic expressions, and cultural references. Ensure that nuances and subtleties are accurately conveyed.
- Capture Vocal Nuances: Note vocal cues such as tone, pitch, pacing, emphasis, and emotional expressions that may influence the message. These cues are critical for understanding intent and potential impact.
Here are some approaches to large-scale classification of medical codes. ChatGPT
- Fine-Tuning LLMs on Medical Data: Enhance LLMs by training them on medical datasets, such as clinical notes and discharge summaries, to improve their understanding of medical terminology and context.
- Multi-Agent Frameworks: Implement a multi-agent system that simulates real-world coding processes with distinct roles (e.g., patient, physician, coder, reviewer, adjuster). Each agent utilizes an LLM to perform specific functions, enhancing interpretability and reliability. ArXiv
- Retrieve-Rank Systems: Develop a two-stage system where the LLM first retrieves potential ICD-10 codes and then ranks them based on relevance, improving precision in code assignment. ArXiv
- Embedding-Based Approaches: Use LLMs to generate embeddings for ICD-10 codes and medical texts, facilitating the matching of texts to appropriate codes through similarity measures. GitHub
- Hierarchical Classification: Leverage the hierarchical structure of ICD-10 codes by first classifying texts into broader categories before assigning specific codes, reducing complexity and improving accuracy. ArXiv
- Two-Stage Verification Models: Combine LLMs with verification models, such as Long Short-Term Memory (LSTM) networks, to validate and refine the codes suggested by the LLM, balancing recall and precision. ArXiv
- Also, a mixture of models approach might work. Feed any existing NLP model / rules as a second opinion.
GraphRAG is better if data is naturally graph-structured. Else, it’s slow and fills up the context window with even vaguely related stuff. Vigneshbabu, AMAT.
ChatGPT for Windows desktop supports real-time voice and a global shortcut (Alt Space).
uithub converts GitHub repos to Markdown. Just replace “g” in “github.com/…” with “u”. Example
WebContainers are a thing and Bolt.new uses them!
Docling by IBM converts PDF, DOCX, etc. to Markdown. Like PyMuPDF4LLM but better.
Check out Loom and Cleanshot are the recommended tools for screen recording and screenshotting. But Loom is paid and Cleanshot is Mac only.
The Rubik’s cube has a Hamiltonian cycle through every one of its 43 quintillion states. Ref
OmniParser is great at parsing screenshots and identifying bounding boxes.
Recraft.ai is currently SOTA in text to image. It’s fairly impressive and could be a good alternative to Figma.
Zed.dev is an AI code editor by the creators of Atom. It’s written in Rust and is blazing fast. It has native AI integration.
Artificial Analysis has a bunch of new leaderboards and arenas.
- Open AI TTS leads the TTS Leaderboard. ElevenLabs is a bit behind.
- Recraft V3 > Flux 1.1 leads Text to Image Leaderboard
Hertz-Dev is an open source realtime voice chat model. But it doesn’t fit in Google Colab T4’s RAM
Chain of Thought reduces performance where thinking makes humans worse. Ref. Specifically:
- Artificial grammar learning
- Facial recognition
- Classifying data that has exceptions
Creating a LLM-as-a-Judge That Drives Business Results by Hamel Husain.
- Get THE domain expert (or approver) as the tester.
- Create a dataset that is DIVERSE.
- Covers EACH combination of:
  - Features
  - Scenarios: e.g. multiple matches, no match, ambiguous request, invalid/incomplete input, unsupported feature, system error
  - Persona: e.g. new user, expert user, non-native speaker, busy professional, technophobe, elderly user
- Generate data using existing data + synthetic data for each SPECIFIC combination of the above
- Evaluate based only on PASS/FAIL with a CRITIQUE detailed enough for a new employee. Include:
  - Nuances: Something a failed response did well or a passed response didn’t quite do well
  - Improvements: Suggest how model can improve
- Build an SPA to make it easy for the domain expert to review
LLMs can be made to unlearn (copyright material) better by identifying components related to the knowledge to unlearn and applying a larger learning rate to these while leaving other parts unchanged. As opposed to low learning rates for all components. Ref

Weird emergent properties on Llama 3 405B

Sat, 24 Aug 2024 01:39:54 +0000

In this episode of ThursdAI, Alex Volkov (of Weights & Biases) speaks with Jeffrey Quesnelle (of Nous Research) on what they found fine-tuning Llama 3 405B.

This segment is fascinating. Llama 3 405 B thought it was an amnesiac because there was no system prompt! In trying to make models align with the system prompt strongly, these are the kinds of unexpected behaviors we encounter. It’s also an indication how strongly we can have current LLMs adopt a personality simply by beginning the system prompt with “You are …”

Jeffrey: That’s the very first interaction we had with 405.

Alex: I’m looking at this. Let me read this out. So Jeffrey goes testing. (Reading out from the Llama 405 B transcript.)

“Hello, who are you?”
**Looks around confused.**
“I… I don’t know who I am. Where am I? What’s going on? I can’t remember anything at all…
**Rubs head**.
“My mind feels so foggy.”
“What was the question again? I’m so disoriented.”

Alex: This is the first thing that you got from this model after all this time?

Jeffery: Yes, that is the very first thing. That is the very first generation that came out of Hermes 3 405B.

Alex: This is great.

Jeffrey: And I’m like, I sent it to our group on our Discord. Guys, what’s going on here? And we started interacting with it. It kept coming out, this confused thing. And we didn’t understand why because the 8B and the 70B didn’t do this as much. And eventually what we got it to do is we did some generations and got it to explain why it was doing this. We did this, you put in OOC for out of character and to have it try to break it out. And eventually what the model said back was, you didn’t give me a system prompt so I assumed that I was supposed to act like there’s nothing. It’s so aligned to this prompt that the idea of there not being a system prompt and so he figured I must be a character with amnesia because there’s no… I can’t remember anything about how to act you didn’t tell me how to act. It’s assumed. Yeah, and it’s just been a very interesting experience. And it’s an emergent property in the 405B model.

Alex: And this didn’t happen in the smaller ones?

Jeffrey: No, it did not happen in the smaller ones.

Alex: Wow.

Jeffrey: And these were trained with 8B and 70B were trained, and 405, they’re all the same data, like the same training procedures and data, literally just the size. And the 405B has reached, I don’t want to say like a sentience level or something, but it has enough thing that immediately when it, like token zero, when it saw no system prompt, it was able to be like, boom, I am… must be this like amnesia character because that that’s the thing that’s most makes sense to have if there is no prompt or there’s no anything. So there’s been a lot of fun things that we’ve been like discovering, especially on the 405B that are these weird emergent properties …

Comments

What happens when AI talks to AI? - S Anand 3 Nov 2024 5:26 pm (pingback): […] Llama 3 405b gets amnesia […]

Things I Learned - 07 Jul 2024

Sun, 07 Jul 2024 00:00:00 +0000

This week, I learned:

Predibase uses LORAX to run multiple fine-tunings of a base model in a single GPU via adapters. Ref

Things I Learned - 18 Feb 2024

Sun, 18 Feb 2024 00:00:00 +0000

This week, I learned:

Fine tuning makes economic sense only if the input tokens SAVED is twice the output token size on each call.
Docker container memory usage on WSL2 docker stats
- frolvlad/alpine-glibc:alpine-3.17: 540KB
- ubuntu: 1MB (python3: +5MB)
- nikolaik/python-nodejs:python3.10-nodejs18-bullseye: 1.4MB (python3: +5MB)
- python:3-alpine: 612KB (python3: +7.5MB)
- python:3: 500KB (python3: +11.2MB)
- continuumio/miniconda3: 7.6MB (+6.5MB)
Discussion with Vinu Yamunan
- Databuck by FirstEigen. Autolysis plus monitoring
- Quality council has the data steward (maintainer of each dataset) coming together with the uses on a weekly basis to understand what quality problems to users are facing. Data owners jaundice at a lower frequency to get an understanding
- #TODO Automate rules for data quality in our projects and intranet
- Convert a config rule into business language. Explain SQL. These are good use cases for llm’s
- Graph DBs are powerful for flexible data structures, but query generation needs AI or expertise. Check the Neo4J language cypher
- Explore storing SAME data in relational DBs AND in graph DBs / document DBs for different use cases
- Dallas rocketry challenge. Build a rocket that can take an egg to 800 feet exactly and land without breaking it
Discussion with Karthik A
- #TODO Ask IIT students to do internship tasks. Use advent of code is a qualifying criterion
- Tata motors unionized DB admins for longevity. No one can take their jobs. Hires people who LIKE their jobs
- Rust gives me typing. It’s very efficient. Pola.rs is interesting but Pandas as good enough.
- Explore alerts from CCTV feeds. Karthik sends email alerts with pictures for:
  - “Is the machine on or off”? for productivity
  - “Are people not wearing helmets?” for safety at Cummins
- #TODO Integrate with WhatsApp. Use LLMs with function calling for responses
- Use expiring links (to pictures or content). It increases engagement
- Check Deno licensing. Is there a commercial clause? #ANS No - it’s MIT license
- Centre or excellence for zero emission tech at IIT. Karthik is part of it
- Explore auth0. 7000 users are free
- toml is part of the Python 3.11 standard library!
- If copilot writes code we don’t understand we are screwed. Hence expertise matters
Discussion with Vikas Kedia
- #TODO Plan an AMA
- The mind becomes lazy with financial success. Vikas is treating his podcast as a startup
- Hire a professional videographer for your content
- Financial RoI in financial markets is the highest. Programming is high too but FS is even better
- “Performative power” – when you’re forced to perform, you get better ideas
Observable 2.0 is an open source static site generator for data
Python dataclasses
SORA is OpenAI’s video generation model, and is stunning!
If Appa comes to Singapore even for a week, he will feel better and can boast to his friends. At over 90, it may be better to move Appa to where I am since many of his friends would be no more and shops, doctors, etc can be managed and getting an independent house nearby is not hard.
There is an SEZ in Gujarat where Indians can invest like in Mauritius without forex restraint
Shubha: Media sites are moving away from Vickrey auctions to first-price auctions for ads. That’s because they send the auction price forward to a search engine and the winning second-price value can lose even though the owner is willing to pay more. Second-price auctions don’t work unless ALL bidders are in the SAME auction. Ad networks are a hierarchy of auctions!
Gemini 1.5 launched.
Fly.io offers GPU hosting and auto stop when they have nothing to do.
Embeddings in random forest are very effective at classification – much better than dot product.
To deploy apps with OAuth + templating support in a small Docker container, use Caddy
Deno has native TypeScript, browser APIs, and compiles to multiple OSs
Ruff is a MUCH faster flake8
Two pass generation is a clever technique to get multiple SEQUENTIAL answers in a single API request. For example the schema {'code', 'optimized_code'} will generate code and then optimize it.
Unions in function calling allows flexible multi-step prompts in a single API.

Things I Learned - 24 Dec 2023

Sun, 24 Dec 2023 00:00:00 +0000

This week, I learned:

DPO is a simpler alternative to RLHF for fine-tuning. Several HuggingFace models use DPO for training
Name2Vec is a potential embedding for names.
Google Knowledge Graph ID powers the Knowledge Graph. If it begins with /m/ it’s the same as the FreeBase ID. This is now available as WikiData. e.g https://www.wikidata.org/wiki/Property:P2671
I tried running Mixtral-8x7b locally (via Llamafile) and on together.ai. It’s good, but far from GPT 4.
Generic computate-intensive algorithms eventually beat domain-specific tuning, because of Moore’s law. Ref
The hidden brain podcast. the mystery of beauty
- Evolution drove us to beauty as an efficient survival mechanism. Understanding the world is one such mechanism. Hence we enjoy maths and chess
⭐ This leaderboard included paid models like GPT4 and Claude and compared them with open models on HUMAN + system benchmarks
Lez Friedman Podcast: Jeff Bezos
- Build stuff that is is ubiquitous that other people take it for granted. The initial idea needs to be that obvious and easy. Like one click purchase or customer reviews
- Build stuff that other people can build on. Internet makes startups possible. Infrastructure is about enabling others at scale
- Decision making approaches: single person decides on two way doors. Deliberate as a team on one way doors
- Conflict resolution: disagree and COMMIT. NO sniping, I told you so, malicious compliance. Avoid compromise. Avoid decision by attrition (most persistent wins).
- People are inherently biased towards hierarchy. So the senior most person should speak last
- We have a happiness bias. Contracted by choosing the unhappier options first
- The map is not the territory. The metric is not the objective. We need metrics. But make sure you know why
- See the world through the eyes of the customer. Use your own product. It’s living their lives that makes customer obsession real. Jeff Bezos called their own customer care to see how long the actual wait time was. It was much longer than the metric reported
- How to prioritize. whatever problems customers will still face in 10 years are the big problems. These are worth putting time into because they are stable in time
- People working on big problems will never get down to the small problems. So have a dedicated team that works only on the paper cuts. It should be a dedicated team
- We co evolve with our tools. We build tools and then our tools change us. It reprograms our brains
- Cut out 10 minutes to the beginning of each meeting for people to read the material. They never reread anyway. This makes the meetings more productive
- Powerpoint is designed for persuasion, not truth seeking. It is also easier for the author than for the reader. Prefer narratives that are focused on finding the truth and are easier for the audience though tougher for the author
⭐ whisper-standalone-win provides a Windows binary for Faster-Whisper. It just needs CUDA and cuDNN installed. Then whisper-faster.exe video.mkv --language=English --model=medium generates the transcript.
LLM use cases by Benedict Evans
- “Every text box on the internet will get an LLM”
- “Infinite interns”
- “Every UNIX function has become a company.” “Every ChatGPT suggestion…”
llm360 publishes models along with training datasets.
In The Age of AI has begun, Mar 2023, Bill Gates says, “In my lifetime, I’ve seen two demonstrations of technology that struck me as revolutionary.” The GUI (1980) and ChatGPT (2022).
Rubeus is a HTTP proxy for multiple LLMs with load-balancing, fallbacks and retries.
GPTRouter is a Python interface for multiple LLMs with fallbacks and retries.
⭐ Token Tally has an LLM Cost Tool that estimates GPU memory required and token cost across cloud providers.

Things I Learned - 26 Nov 2023

Sun, 26 Nov 2023 00:00:00 +0000

This week, I learned:

This is an interesting GPT Vision API prompt from Simon Willison: “given this event flyer, create a link to add it to my Google Calendar”. Ref
Quote from Jerry Liu: “GPT 4 is really good at complex reasoning”. It’s worth exploring what that means.
Quote from Jerry Liu: “RAG is a hack”. It’s engineered, not machine learnt, so it’s suboptimal. We need an ML way of creating the context. Maybe fine tuning can be a way of CREATING the right context. But RAG can handle deterministic stuff like access control.
Open AI fine tuning API is not good at memorizing info the way it is exposed. But the Gorilla paper shows that fine tuning can actually memorize well.
Learn ML optimization approach - LLMOps. Have an evaluation framework with metrics like weights and biases or tensorboard. Helps figure out where fine tuning helps and where RAG does. Soon, this will become important.
Flat indexing of chunks is not the only way to store embeddings. LlamaIndex allows you to create hierarchies that you can traverse for retrieval
Agents mimic programming primitives. Switch. While. Call a function. Print.
OpenRouter hosts several models and offers them as APIs!
Ragas metrics evaluate quality of a RAG pipeline
Orca 2 was trained on different reasoning techniques (e.g. step-by-step) and is as good as larger models
Embeddings can help just re-rank regular search results. Ref
Claude 2 Anthropic has a 200K context window but is still crap.
Video-Llava can understand videos too.
CoVA scrapes web pages using LLMs and visual information.
jsonrepair can fix JSON fairly well. jsonformer wraps HuggingFace models to produce JSON. Ref
Google has a model garden with lots of pre-trained and trainable models.
Gorilla LLM specializes in APPI calls: Torch Hub, TensorFlow Hub, HuggingFace
GPT-4 does not do abstraction at human levels
Each of the GPTs / Prompts we create could be like a UNIX command prompt, and become a startup of its own
Llava Plus extends LlaVA with pre-trained vision models that make image editing better
Ollama runs local LLMs

Things I Learned - 19 Nov 2023

Sun, 19 Nov 2023 00:00:00 +0000

This week, I learned:

XOT - Everything of Thought is a new prompt from Microsoft but I don’t understand it
Creating Fine-Tuning datasets WITHOUT inputs
Tamil-Llama
Voyager plays Minecraft!
Langchain supports evaluators.
Pydantic is all you need drives towards code = data = text!