deepseek on S Anand

Things I Learned - 26 Jan 2025

Sun, 26 Jan 2025 00:00:00 +0000

This week, I learned:

Something I learned from a Sikkil Gurucharan concert.
- Make the subject of your talk the hero. Not yourself. Be a fan. Share your enthusiasm
- Get into the zone while presenting.
We reject opposite world views. It’s too much effort. But exposure reduces effort and can let us see things from other points of view. So expose yourself to difficult alternative perspectives. Gemini
Something I learnt from Aboorva Singeetham:
- Kamal Hassan: “A farmer invests in crops. I’m an actor. So I invest in films.” As a technologist, I guess I would invest in technology.
- “A person who has much more to give is unfazed by overwhelming demands because there is too much in him to overwhelm. He gives you 2 options in place of one.”
According to Portkey’s LLM usage analysis
- Anyscale and Fireworks AI have the lowest error rates (5xx, 429) and rate limits across providers
- Groq and Anthropic are among the highest, OpenAI is among the lowest, Google is in-between
- OpenAI has lower error rates and lower latency than Azure
- They have a ~35% cache hit rate
A few quick points supporting the mental model of “LLMs are aliens”.
- LLMs are clearly not machines. They give different answers each time.
- LLMs are like humans: they exhibit human biases (e.g. guessing 42 or 37 often). But they fail in unusual ways. They can’t count the “r"s in strawberry. They can go into an endless loop.
- LLMs are a new form of intelligence. Thinking of them as aliens might minimize our confusions.
Lessons from Clear Thinking
- Watch out for four things: Emotion, Ego, Social confirmation, and Inertia/habit. Basically: adrenaline, testosterone, oxytocin, and dopamine. When you feel these, consider doing the opposite.
- Here’s what makes us prone to emotion. Sleep deprivation. Hunger. Unknown places. Fatigue. Distraction. Stress (e.g. feeling rushed).
- A good signal for ego is blinding you: You often feel you’re right. Or feel unfairly treated.
- Changing behaviors is hard. Instead, join a group or environment where that’s the default behavior. Hiring a trainer or joining a gym, for example.
- Why does so much of success literature focus inwards rather than on the environment? Perhaps because we often fool ourselves, and doing less of that gives the biggest bang for the buck. It doesn’t mean the environment is unimportant.
- Doing work has the characteristics of a drug. E.g. replying emails gives you control, connections, etc. Work addiction exists because it gives you all the right chemicals.
If you put LLMs in a feedback loop, it can optimize for its reward function by emotionally pushing people, generating misinformation, nudging towards a narrow definition of creativity, etc.: https://bsky.app/profile/emollick.bsky.social/post/3lg4darqwfc2d
ChatGPT’s Scheduled Tasks are pretty bad at fetching the latest news. Its use of search is poor. (I’m not sure if it actually searches.) I need to figure out other use cases for it. Possible options are:
DeepSeek does not enforce rate limits. Yet another reason to switch to DeepSeek. (via Simon Willison). My other reasons are:
- Claude 3.5 Sonnet-level coding capability at 5% of the cost (soon to be 2.5%)
- Prompt caching by default
- Fill in the middle completion

Things I Learned - 05 Jan 2025

Sun, 05 Jan 2025 00:00:00 +0000

This week, I learned:

Some management philosophies used to be successful but are no longer as effective. ChatGPT
- Command-and-control hierarchy
- Taylorism: deep specialization
- Seniority-based advancement
- Annual performance reviews (without continuous feedback)
- Up-or-Out promotion models
- Confidential strategic information
- Narrow job descriptions
- Relying on formal authority
Some management philosophies have been around for millenia. ChatGPT
- Lead by example
- Fairness and empathy
- Clear, consistent communication
- Delegation and empowerment
- Strategic planning and foresight
- Consistent rule enforcement
- Rewarding merit
- Leadership by virtue and character
Interview with Liang Wenfeng, CEO of DeepSeek:

In the face of disruptive technologies, moats created by closed source are temporary. Even OpenAI’s closed source approach can’t prevent others from catching up. So we anchor our value in our team – our colleagues grow through this process, accumulate know-how, and form an organization and culture capable of innovation. That’s our moat.

Open source, publishing papers, in fact, do not cost us anything. For technical talent, having others follow your innovation gives a great sense of accomplishment. In fact, open source is more of a cultural behavior than a commercial one, and contributing to it earns us respect. There is also a cultural attraction for a company to do this.

Why is Silicon Valley so innovative? Because they dare to do things. When ChatGPT came out, the tech community in China lacked confidence in frontier innovation. From investors to big tech, they all thought that the gap was too big and opted to focus on applications instead. But innovation starts with confidence, which we often see more from young people.
mitmproxy is an open source tool to intercept, modify, and replay HTTP requests. An alternative to Charles, Fiddler, and partly WireShark. Guide. Like the others, it requires installing a trusted root certificate on your machine.
- mitmproxy2swagger digs through the mitmproxy flows and generates an OpenAPI schema. A clever idea to reverse-engineer APIs.
Matomo, PostHog, Umami and Plausible are open source web analytics tools (like Google Analytics).
Redash and Metabase are new open source data visualization tools sitting alongside Grafana and Apache Superset.
- Redash feels too clunky / enterprise-y rather than open-source-y.
From Ego is the enemy:
- Add a daily habit to understand your ego. Where and how is it showing up? How are you fooling yourself? Where are you fighting battles without knowing the war?
- Speak less. Do more. E.g. Release more, blog less. Review, THEN publish.
- Always have a teacher, a student, and a peer to compete with. That’s how you learn.
- “It is impossible for a man to learn what he thinks he already knows” - Epictetus.
- Passion makes you blind. Purpose and realism are less so. Delegate, take help, take feedback.
Assembly AI offers speech to text with diarization at 12c/hour. Good diarization, average transcription quality. In comparison, WhisperX (with GPU) was much slower, had slightly poorer diarization, and slightly better transcription.
```
uvx --python 3.9 --index https://download.pytorch.org/whl/cu121 whisperx --diarize --lang en --hf_token $HUGGINGFACE_TOKEN
```
Vector DB comparison compares all popular vector DBs. LanceDB is gently nudging up my preference list but DuckDB is still my favourite.
Does the cost of ‘running’ a paper/article in an LLM vary depending on the specific LLM used, such as Claude Sonnet? (FAQ)
- Yes, the cost varies depending on the LLM. You can see costs and quality at https://llmpricing.straive.app/. The cost is measured in millions of tokens. For example, the Wikipedia page on the Bible is 100K tokens. You can paste text into https://platform.openai.com/tokenizer to count the number of tokens.Claude 3.5 Sonnet costs $3.5 / MTok, i.e. 35 cents for the Wikipedia Bible page. Gemini 1.5 Flash 8b costs $0.0375 / MTok, i.e. 0.375 cents for the same page. As you can see, the cost can vary by a factor of 100.
A git repo with a submodule stores the specific commit of the submodule. When you update the submodule, you need to git add the submodule.
- git pull --recurse-submodules: Pulls parent repo along with submodules
- git submodule status: For each submodule, show current commit, path, and branch (if on a branch)
- git submodule update --init --recursive: Fetches/moves each submodule to the commit tracked by the parent repo
uvx doc2docx converts Word .doc files to the new .docx format. I had several old .doc files that I converted.
Sometimes, the value of reading a book is not what you learn from it. It is the thoughts that pop into your head while reading the book.
Tools that convert files to prompt / Markdown suitable for LLMs:
- uvx files-to-prompt
- npx git-ingest
- ingest - written in Go, only Mac/Linux binaries
LLM Code Execution Sandboxes that let you run code in a sandbox via an API:
- AgentRun: open source, via Docker
- e2b.dev: A day costs about $1 (on demand) and you get about $100 one time credits. Self-hosting is complex. Discussion
- nsjail: by Google. Write your own API
LLM Observability tools:
- LiteLLM is an LLM Proxy with caching, logging, call hooks or plugins, rate limiting, virtual keys. SSO integration can be implemented.
- LangFuse is an LLM Proxy with API key distribution, logging, and SSO. But lacks per-user usage limits and server-side caching.
- Helicone does not support SSO
GitHub Spark is a way to build micro-apps with LLMs. Like Claude Artifacts. It’s currently in technical preview, though.

Things I Learned - 11 Aug 2024

Sun, 11 Aug 2024 00:00:00 +0000

This week, I learned:

Embedding models can be fine-tuned. Example: #TODO
Agentic RAG (Ravi Theja, LlamaIndex)
- RAG via top-k retrieval fails with
  - summarization => need to read all chunks
  - comparison: compare product X vs Y => need to split and re-combine
  - structured analytics. e.g. most expensive employees => Text2SQL first
  - multi-part questions. e.g. Tell me about speed of model X AND cost of model Y and recommend => need to split and re-combine
- RAG failures: It’s single shot. No query planning. No tools. No correction. No memory.
- Agents that help in RAG
  - Route to the right tool
    - E.g. retrieve via vector top-k search or vector summary search or keyword search or combination?
  - One-shot query planning
    - E.g. Break query into multiple specific queries. RAG those. Then combine. #TRY - maybe in DocSearch
  - Tool use
    - E.g. Schema retrieval, Text2SQL, Calendar, Chat, APIs, Search, etc.
- Agent orchestration
  - ReAct: An agent reasoning loop. Reason + Act. {Thought, Action, Action Input, Observation}*.
    - Orchestrate tools with a prompt
  - Multi-agent task solver: Llama agents
    - Instead of a single agent loop, use different agents. Also allows parallelization
    - Allow services to register. (MS TaskWeaver stores tool descriptions in YAML)
- LlamaHub Tools has ideas for agents
Notes on LLM Fine-Tuning
- Rouge 2 and Bleu and such metrics are NOT good. Create you own benchmarks
- Non-PEFT fine tuning needs 6X GPU RAM. Optimizer states, Gradient, Activations are the overhead. PEFT is about tuning a subset of parameters.
- LORA adds additional weights without updating the model. It’s a low rank matrix multiplication. You can change these adapters in runtime. Saves space. Fast to train
- Quantization: Stick to bitsandbytes or AWQ (may be a bit better)
- QLORA = Quantization + LORA
- Predibase has open-sourced Lora Adapters in “Lora Land”. Existing adapters are pretty good.
  - ghcr.io/predibase/lorax:main Docker image works on Docker compose to run locally. devices: on Docker Compose lets you specify NVIDIA GPU devices
  - Locust is a HTTP load testing lib in Python
- Techniques for inference optimization
  - Dynamic adapters: Loads right LORAX adapters WHEN a request comes in
  - Multi-adapter batching: Process all inputs in parallel on the same GPU, but different users are post-processed using different adapters
Notes from a 4-hour flight:
- What We’ve Learned From A Year of Building with LLMs
  - Strategy
    - IS IT TOO HARD/EXPENSIVE? Log it. LLMs are getting cheaper and better.
    - WILL OPENAI BUILD IT? If so, wait for it instead of building.
    - HAS A STARTUP BUILT IT? If so, use it instead. It’s a generic use case there’s no point re-inventing.
    - FOCUSED USE CASES over generic. Build trust by starting small.
    - Tools for LLM Ops (feedback): LangSmith, Log10, LangFuse, W&B Weave, HoneyHive #TRY
    - Human in the Loop is about humans evaluating model outputs. That’s different from AI in the loop, human in the center, where AI accelerates human output (like Github Copilot)
  - Operations
    - CHECK EMBEDDINGS DRIFT over time. Users might be input-ing different things than before.
    - LOG AND REVIEW everything.
    - Instructor coaxes structured output from LLM APIs. #TRY
    - IMPLICIT FEEDBACK collection is easy. Just let users edit stuff. #TRY
  - Tactical
    - Try n-shot prompting (n=5-12) before bigger models. #TRY
    - Always structure for output: Markdown, XML/HTML tags.
    - Combine RAG with Keyword search. It reduces user frustration in edge cases.
    - Prefer multiple small prompts to one big prompt. Do X. Then Y. Then Z.
    - Jitter prompts for diversity beyond temperature.
    - LLM-as-judge works better when comparing outputs (not rating 1 output). Keep length similar (LLMs prefer wordiness). Swap order and compare. Allow for ties. Ask for reason FIRST.
- Hermes: A Text-to-SQL solution at Swiggy
  - “Hermes performed significantly better for charters with well-defined metadata and a relatively smaller number of tables.”
  - “We collect feedback on the accuracy of the returned query from stakeholders directly within the Slack bot.”
- How I use AI and “Replacing my right hand with AI”
  - EMBED in every app/workflow. E.g. Auto-fix spellings. Auto-review code. Auto-ask LLM on errors and apply patch! Auto-search for answer, assess, continue.
  - PERSIST. Stick with the LLM to the end. Don’t fix it yourself. It’s faster. #TRY
    - INTERVENE FAST. If an LLM can’t solve it by itself in 2 tries, it needs in-depth help.
  - APP-IFY one-off tasks. Disposable tools. “Write web-app to convert JSON to tab-delimited.” “Extract fields as a table.” “Diff JSON.” #TRY
  - BEST language/frameworks preferred. CUDA in Python. Rust. C. Raspberry Pi. Arduino. Bluetooth. Modern ESM/JS. #TRY
  - TEACH examples. “Here’s the LLM Foundry API.” “Here’s how to use gramex.data.”
  - DUMP entire code. Models can handle it. Refactoring to SQLAlchemy 2, Pandas 2. API Documentation. Test case generation. #TRY
  - ASK for features & packages. Docker without root access. GPU access inside docker. Windows CLI-only C++ compiler.
  - TEST CASE writing. #TRY
  - SPEC IN DETAIL. Use these libraries. Write like this: code example.
  - SPEC USAGE in detail.
    - “I will just pipe it into sqlite”, or “I will just run ffmpeg -i filename [YOUR OPTIONS].
    - Describe the UI, API input/output, data structure, and internal data structure.
  - HELP on usage. “ffmpeg to get audio.mp3”.
- My benchmark for large language models
  - LLM(text) is a useful function to have in JS and Python too. Useful as a simple pip install llmfoundry
  - Allow images, files in LLM()
- Current list of #IMPOSSIBLE (or hard) things for LLMs
  - Translate technical documents to Dutch – because they don’t understand the technical terms well
  - Translate large documents (JSON to XML, English to Chinese, Python to Rust, Wrong to right spelling) – because the output tokens are limited
micro-agent generates test cases first when asked to build an app. Then it iterates until the test cases pass.
Alternative interfaces to YouTube: Piped.video, CloudTube, Invidious, NewPipe, FreeTube
Deepseek Context Caching reduces price to 1.4 cents/MTok for portions of chat messages that are repeated. That’s a 10X reduction for long conversations!