gpt-4o on S Anand

Sun, 27 Apr 2025 09:36:39 +0000

This is my decision tree for which model to use on #ChatGPT right now.

O𝟯: Use by **default.
O**𝟰-mini-high: Use when **coding.
GPT** 𝟰o: Use for a quick response or to create image.

Things I Learned - 01 Dec 2024

Sun, 01 Dec 2024 00:00:00 +0000

This week, I learned:

Gists are a good place to store static files for posterity as well as throwaway files. But, they’re just git repositories. So there may be no advantage over GitHub repos.
GPT-4o Audio supports tone control via XML tags like <cough>..., <laugh>..., etc. But at ~$15/hr of output, it’s too expensive. Ref
Mridula’s son gave a live commentary of what he was doing on Minecraft and ChatGPT gave him live evaluation and coaching. E.g. “Great strategy! Getting to the launch pad early can give you a huge mobility advantage. Making the bridge wider is also a smart move to prevent accidental falls. With this plan, you’re setting yourself up for success. This is a great way to interact with LLMs.
Gemini’s JSON mode returns JSON with keys in alphabetical order. I think. Emperical evidence. This is unlike OpenAI which explicitly returns the keys in the order specified.
- To solve this, order the keys alphabetically.
HTMX focuses on HTML over JS. Like server responses being HTML snippets not JSON. But I need front-end over back-end. Client side apps. HTMX doesn’t help much there, e.g. templating, or just plain JS code.
- htmx client side templates do can convert JSON to HTML.
I installed the OpenAI Desktop App as well as Claude for Desktop. They take up too much RAM (260MB and 750 MB respectively on startup - though this varies.) The ChatGPT web page takes ~100MB incrementally, so I wrote an AutoHotkey script to switch to the first open (or recently closed) ChatGPT tab on Brave.
I tried LIDA from Microsoft, after almost a year of its release. A few notes:
- Just running uvx lida ui --port 8080 --docs works.
- But I needed to use export TCL_LIBRARY=C:/Users/Anand/AppData/Roaming/uv/python/cpython-3.13.0-windows-x86_64-none/tcl/tcl8.6 to point it to my TCL installation for charts to work. I also chose to export OPENAI_BASE_URL=https://llmfoundry.straive.com/openai/v1
- I also chose to replace gpt-3.5-turbo-0301 (the default model) with gpt-4o-mini in lida/web/ui/component*
- It’s quite impressive.
OpenAI allows multiple system messages. I learned this browsing through the LIDA prompts.
Anthropic’s Model Context Protocol lets any apps integrate with LLM Apps. LLM Apps are becoming the new operating system. Competitors, beware.
I spoke at Automating Data Visualizations using LLMs at SUTD. Apparently, using LLMs to write code is much more common than writing code to use LLMs. I ran a quick quiz.
- Have you used ChatGPT or any LLM? 35 / 35 raised their hands.
- Have you written code using an LLM? 34 / 35 raised their hands. (I was impressed.)
- Have you uploaded a spreadsheet to an LLM for analysis? 15 / 35 raised their hands.
- Have you programmatically called an LLM API? 6 / 35 raised their hands.
With LLMs, fostering innovation is a new path to profitability. Companies are increasing innovation team sizes. Productionizing that is the next. Some initiatives are:
- Convert popular demos into starter kits
- Create and evangelize trainings on solutions and solution techniques
- Create larger pools of capacity to build innovation and productionize it
Andrew Ng Explores The Rise Of Al Agents And Agentic Reasoning | BUILD 2024 Keynote
- Innovation is now a path to production. People are able to build 20 prototypes at the cost of one and see which sticks
- Machine learning is much faster. Things that took months now date days. But engineering and evaluations are only slightly faster and have become a bottleneck
- A good analogy to zero shot prompting is to ask a person to write an entire essay without pressing backspace even once
- Andrew scenes to align with the line chain definition of agentic workflow, which is about agents being able to craft their own control flows
- People find it very easy to understand agentic workflows once they read through the code
- Reflection or feedback is a useful agentic pattern
- In multi-agent collaboration, it may be the same underlying model that is acting as different agents. But just like we find it useful for the same CPU to run multiple processes and each application is its own abstraction, agents of useful abstraction
It’s hard to summarize a large document using RAG. But you can directly add answers to such questions into the corpus, e.g. by adding a “summary” section, and other answers to common questions.
CloudFlare workers can bundle any kind of files, including text, data, and WASM. Docs
AssemblyScript can compile TypeScript to WASM. Here’s what I learnt
Here’s a convenient pattern to git commit a directory but nothing else in it (e.g. a build/ directory). Add a .gitignore file with * followed by !.gitignore. Only the .gitignore file is tracked.
Ultravox lets you build voice agents at 5c/min = $3/hr (OpenAI is 6c input, 24c output). Or clone their repo.
- Idle call time is counted towards cost. So cost may be higher than OpenAI.
- Voice cloning quality is average. Very distinctive voices are just partly identifiable.
- Supports tool calls (from their server).
- Their API is simple but the docs have minor errors (e.g. a trailing comma in the JSON, which leads to an error) reducing confidence.
LLMs may be good at derived data generation. For example, given a database schema, what derived columns would be useful? What derived views would be useful?
The O1 model does not have a mechanism to control the amount of tokens to spend on reasoning. DeepSeek R1 might, but the API is not out yet.
The OpenAI Desktop App can interact with native applications, e.g. read from Terminal, VS Code, etc. This takes it on a path to becoming a copilot for ANY apps. Putting every copilot app and every LLM integration under threat.
Crawl4AI and Firecrawl are tools / libraries to convert websites into LLM Friendly Markdown and extract structured data using LLMs.
Don’t try and solve specific problems. Pass the entire context to an LLM and get a comprehensive solution. Most doctors, for example, ask specific search-like questions instead of uploading the entire case history and asking for a diagnosis, and perform workse than LLMs. Ethan Mollick

Sat, 10 Aug 2024 10:59:35 +0000

Fascinating to see the how LLM cost-quality frontier moves. Recent fights were mostly on cost.

Yesterday, #OpenAI halved the GPT-4o cost. At $2.5/MTok (and with GPT-4o-min at 15 cents/MTok), the best and cheapest models are back with OpenAI, IMHO.

Sigh, time to move all our stuff back from #Anthropic. For now…

https://gramener.com/llmpricing/

Mon, 22 Jul 2024 03:19:27 +0000

Loved this Rocky Aur Rani Kii Prem Kahaani scene where Ranveer asks, “Chinese ko Chinese bol sakte hai?”

हम बहनदी भी नहीं बोल सकते? आंटी, मैं दिल्ली से हूँ। मैं कैसे नहीं बहनदी बोलूं बहनदी!? कैसा जमाना आ गया है? फैट-ों को फैट नहीं बोल सकते, ब्लैक-ों को ब्लैक नहीं बोल सकते, ओल्ड-ों को ओल्ड नहीं बोल सकते, मुँह खोलने से डर लगता है मुझे! आप मुझे बताओ, चाइनीज़ को चाइनीज़ बोल सकते हैं?

Can’t we even curse “Damn it!”? Aunty, I am from Delhi. How can I not say, “Damn it”, damn it!? What times are upon us? You can’t call fat people fat, you can’t call black people black, You can’t call old people old, I am actually afraid to open my mouth! Tell me, can you call Chinese people Chinese?

अगर मुझे चाइनीज़ आर्डर देना है तो क्या करूँ? फ़ोन करूँ “हाँ भाईसाहब, वह हिमाचल से राइट जो जगह है, हाँ हाँ पड़ोस वाला मुल्क, हाँ वही कोरोना वाला, हाँ, वहां का एक क्रिस्पी चिकन लगा दो” अब आप बोलोगे “कोरोना वाला” नहीं बोल सकते!

If I want to order Chinese, what should I do? Should I call, “Yes bro, that place to the right of Himachal Pradesh. Yes, yes, the neighboring country, yes, the one with Corona, yes. Get me a crispy chicken from there” Next, you’ll say, “You can’t say “Corona”‘!

बचपन से, बचपन से दादी कहती आ रही है, “चाय पीने से काले हो जाते हैं, चाय पीने से काले हो जाते हैं”। अब पता चला है, ये रेसिस्ट है!

Since childhood, Grandma taught us, “Drinking tea turns you black. Drinking tea turns you black”. Now I learn that’s racist!

किसी ने ये भी नहीं सिखाया कि कौन सी गाली देने से मिसोजनी हो जाती है। यह भी नहीं बताया कि गोलू को गोलू बोलने से फैट शेमिंग हो जाती है। हमको तो यह सब नॉर्मल लगता। हमको तो कभी रॉन्ग लगा ही नहीं। हमको पता ही नहीं यह बोलने से सामने वाले की फीलिंग्स हर्ट हो सकती हैं!

No one ever taught us which abuses become a “misogyny”. No one ever told us that calling Chubby ‘Chubby’ is “fat-shaming”. We all thought this was normal. We never had an inkling it was wrong. I didn’t know that saying this may hurt others’ feelings!

Beautifully worded. Lovely acting. Nothing I could add to it.

(Except: Whisper and most transcription APIs handle Hindi poorly. Claude 3.5 Sonnet does better than GPT-4o on Hindi grammar.)

Things I Learned - 09 Jun 2024

Sun, 09 Jun 2024 00:00:00 +0000

This week, I learned:

httpretty can mock ALL Python HTTP libraries
Japanese pray to dead parents instead of gods. The dead are preserved in plates by priests. Japanese are generally non religious
Looks like GPT-4o is using CNNs to create vector embeddings of images, with images gridded into a 1x1, 2x2, etc. PLUS OCR. Ref
The sum of a sinusoidal series is like a spirogram. Spinning circle linked to another and so on https://www.andreinc.net/2024/04/24/from-the-circle-to-epicycles

Thu, 09 May 2024 07:36:35 +0000

There are 4 frontier #LLMs today. No other (popular) model beats them on BOTH cost and quality.

llama-3-8b-instruct
claude-3-haiku-20240307
llama-3-70b-instruct
gpt-4o-2024-05-13

This list changes rapidly. But in practice, it means there’s little reason to use any other LLM. They beat every other model on cost and quality (measured by the LMSYS Arena ELO score.)

I opened Straive + Gramener’s keynote yesterday at marcus evans Group’s Digitech forum with this. Strange that this is not well known. Especially as switching from GPT-4 to Claude 3 Haiku can shrink a $1.2 million Gen AI budget to just $10K.

See the interactive version at https://sanand0.github.io/llmpricing/

10 May 2024: mistral-7b-instruct-v0.2 was dropped since llama-3-8b-instruct is available for cheaper on Replicate.
19 May 2024: gemini-1.5-pro-api-0409-preview and gpt-4-turbo-2024-04-09 were dropped since gpt-4o-2024-05-13 is half the price at similar quality

Things I Learned - 21 Apr 2024

Sun, 21 Apr 2024 00:00:00 +0000

This week, I learned:

Effort engine introduces “effort” as a parametrizable way to speed up LLMs with a quality trade-off. Works on Mistral for now.
Many arts demand devotion. Devoting unrestricted time is part of that. 16 hours of practice a day is not uncommon. Sessions don’t start and end on time.
Instruments take a lot longer to learn than vocal music. The instrument needs to become an extension of you.
Tests and homework have a purpose. It helps people figure out whether they’ve learnt. So:
- Write tests that make people think! Like DuckDB workshop
- Share a list of exercises that people can explore
People need to explicitly be INVITED, and potentially IN PERSON, before they will engage with something new.
- For example, no one posted to GenAINews@straive.com until the VIA Talks session where we got them to post.
- For example, having one day at IITM mandatory (especially early in the course) gets online students familiar with TAs. They understand that TAs actually help, at high quality. That they can use Discord.
- What makes Delhi students more assertive? How can we inculcate that in others?
jsr-io/migrations is a great example of database migrations.
Shape Detection API in the browser detects QR codes, face bounding boxes,
Browsers also natively support blurring and face tracking. via
Lessons after half a billion GPT tokens for GPT-4:
- Vague instructions are better than over-specifying
- Avoid libraries like Langchain. APIs are stabler
- 1 token = 3 characters is good enough
- GPT4 doesn’t hallucinate much, except it does a poor job of saying “I don’t know” or “There’s no such data” (the null hypothesis)
- Keep the output down to 10 items or so if you’re listing. For longer lists, have it explicitly enumerate
- Don’t worry about niches. Just wait for GPT5
#WRITE GPT clearly prefers 42 as a random number.
#WRITE fal.ai “animates” pictures, creating videos. It made one from my talk. I morphed into various somewhat similar people rapidly in a 2-second span. Very promising, and far from good.
llmsherpa extracts PDFs using LLMs. It has errors but it preserves hierarchy, extracts tables well, and retains image coordinates. Via +91 90031 35354 ~Vetrivel PS
www.web.sp.am is a content farm that’s getting hit by OpenAI. Highlights how easy it is to create content farms, and therefore “easy” it can be to introduce bias into LLMs.
OpenAI supports batching requests. Didn’t know that.
Marvin provides Python decorators to create AI functions. Pretty intuitive!
Outlines generates structured test with LLMs. It uses the ⭐ logit_bias trick to limit choices in output. See get_choice()
Lemur from Assembly.ai does real time call transcription and summary
W3C is exploring ways to allow web pages to train LLMs, to flag content as AI generated, etc.
Data Provenance Explorer lists open datasets used to train LLMs.
Summarize.tech summarizes YouTube videos.
#WRITE Stable Audio 2.0 generates 3 min of music from a prompt. I tried Bollywood Tamil film background music. Dark, soulful and Horror movie background. Drums starts darkly. Build up to a crescendo of intense chaos.. Great that it managed, but not great music. Somewhat stereotyped. I need to learn how to prompt better. BTW, Udio is another such.
Harpa.ai is a well designed Chrome extension / plugin that can chat with or automate any page.
Due to in-context learning, giving 100s of examples in the prompt can teach LLMs to jailbreak. Ref
With RAG on search becoming big, search APIs are growing. serper.dev, you.com, searxng being examples.

Things I Learned - 07 Jan 2024

Sun, 07 Jan 2024 00:00:00 +0000

This week, I learned:

Raman Srinivasan:
- IITM Profs and MTechs are spinning off deep tech startup.
  - Agnicool is an example. They 3D print rockets with ceramic composites from Germany
  - Sriram Krishnan (Facebook), Balaji Krishnan invested in pre-Series A
- Govt is de-regulating space tech and geospatial. Talking of de-regulating nuclear.
  - ISRO seems to be focusing on cutting edge while others are doing commercial stuff
  - There are about 100 space tech startups in India
  - You can build your own modular reactor
- Geospatial AI is a big opportunity
  - Have released a lot of 10m resolution geospatial data almost for free
- success is about getting NO factor wrong. Failiure just requires one aspect to fail. Brand, business savviness, financial stability, tech superiority, deep pockets, managing Gvt, long-term mindset, etc. - all of these matter. That’s what made TCS monopolize the exam business in India.
- For deepening AI, we need, Talent, Data pipelines, Hardware
- Next wave is LMMs, not LLMs
- What’s not captured in LLMs is verbal knowledge and tacit knowledge (in people’s fingertips). India is rich in this. The road to tacit knowledge has to go through India
  - We can get a welder to train a simulator and pay the welder
  - We can get a storyteller to tell a few stories and train oral LLMs
- Tacit knowledge will have to cover robotics. Train robots to bring coffee in just 50 demos!
- “Project delays are within the ‘rulebook’. Buyt paying skilled welders for ship building or nuclear pressure boilers needs breaking 100s of rules. Once they get certified, they abscond to Iran or somewhere.”
- TCS Ignite started in 2006 by Ramadorai. Before recession. “There is going to be a talent shortage. Recruit from next rung. Science not engineering graduates. Break HR monopoly and corruption - colleges became placement agencies. Fewer people per college. Across the country. Train them.”
  - Tried in 2000. HR refused. Business refused. When Chandra was identified, Ramadorai took it up himself as a challenge.
  - Ramadorai had very precise attention. Sat 7 am calls. “What are you doing?” 2 min call. Enough to energize.
  - Would exchange and ask for brief updates. He reads and responds. You get a decision in a few hours early in the morning. No decision bottleneck
  - He wanted to know ALL the details. Very precise, small, frequent probes on what’s happening.
  - E.g. one 6 am, he called. “What are the lectures planned for today?” He expected I would know this. If not, next time I would be prepared.
  - He would call another person and ask the same question. So I updated the others.
  - I’ve never seen anyone with that bility to ground-truth.
  - He wants 10 birds from 1 stone. Get BSc, but don’t comprimize. Get the best 2 per college but a full batch size of 500.
  - We became the biggest training program as a single batch – with 500 people. He wanted to demonstrate scale.
  - HR and CFO said, ‘You recruit first. Then we’ll give you money. We don’t think it’s possible."
  - We had anchor colleges and brought people from other campuses.
  - We did digitized exams. Took big servers to the campus. Fully digitized with full auditability. Plugged the laptops into the college LANs. Kids had never used a mouse. We had to teach them. We said, “Don’t worry. These are logical questions, not questions. We’ll pay a full salary.”
  - We learned that 1 out of 2 didn’t even join. Many took up a Masters. They didn’t want to join the workforce. Unless they’re desperate economically. Even poor parents, if they can afford to support you at home, they do that. It’s weird.
  - Every weekend, we visited a few campuses. 71 locations across the country.
    - Found the NSS college in Ottapalam (Kumbakonam of Kerala. Cultural centre.) College had a nice nice Math dept website. I said “Mr Ramadorai, this looks promising.” One Sat morning, he called and said, “When are you going to Ottapalayam?”
    - We landed in the college. There was an impromptu communist student strike. We made 38 offers out of 100 who took the exam. Never had such a high conversion. One girl, whose father was a coolie, jad communication issues. Had a colleague talk in Malayalam. She was an amazing success. My colleague Murali made a documentary about her.
  - We started in July. By Dec, we had 500 joinees.
- No one is doing such a thing now.
  - You have to get dozens of things just right. Compromising on even one kills it.
  - Ramadorain loaded it with multiple objectives. Fresh talent. Low cast. Sustainable.
  - He kept pushing for innovation. I pushed back. But he was persistent. Over time, I came around and we started innovating.
  - We restructured training program around innovation. Like a YCombinator. That unleashed extraordinary energy. Several of the kids are running their own startups. Ramadorai was very supportive of that.
  - The assessment product came out of that.
  - First batch, everyone was very sceptical. We got a lot of pushback. They’re dumb. Ethics issues. Communication issues. Lot of prejudice. So we got them to do internal recruitment till they were satisfied. An internal placement market. THEN reputation was set.
  - I told them, always stick to the dress code. One weaver’s sone wore a bright yellow polyester T-shirt. I asked him why he didn’t stick to the dress code. “Sir, it’s my first T-shirt.”
  - Ramadorai tracked how many became billable. We were unable to place 70. He said, give them 1 more month training. Then we placed 64 of the 70. He said “Do something about the 6. I want 100% placement.” We absorbed them as a teaching assistant. One was a weaver’s son. One was a PC’s daughter. A mestri’s son. A shopkeeper’s daughter from North Madras. None could speak English. They learned to code and helped build the exam software, with Srikumar who was a brilliant Java coder. That gave us the confidence that these are good kids, just from the wrong part of town. With a good guide, they’re very capable.
- We bought a bunch of Nintendo Wiis. Kids have to play.
  - He asked for a welding simulator. “Velu the Welder”. The kids built it using the Wii.
  - We got the most accomplished welder spend an afternoon at Ignite. He ripped us apart. 4 hrs non-stop. He told us EVERY thing wrong with it. Blasted us.
  - I told Murali, “Let’s call it a toy. It’s not a simulator. Let kids play.” He said, “I want to show that it can be done!”
  - Murali churned out rapid iterations in a frenzy.
  - Ramadorai said, “Deploy it in the field.” So we went to all kinds of remote places like Gondiya below Nagpur. Surprisingly cosmopolatan. Junction of EW and NS train lines. We set up welding institutes in each. It was on the cloud. We could track everything.
- KPK killed the skills. Hard core bureaucrat. His view is colonial.
  - Ignite philosophy is about unleashing energy of people.
  - Colonoial model is about controlling people by keeping them poor. KPK and Chidambaram had that mindset.
  - Ramadorai brought him in as Secy of NSDC. he killed the policies
  - Modi did the first cut by creating a ministry. KPK ensured that it never gew. Like Yes Minister. Made sure nothing moved
  - Had Govt not changed, he would have been Secy Finance. He was seen as Chidambaram’s blue eyed boy. People know he was associated with NSE scam. Ramadorai helped by bringing him into skills
  - He is very smart. Knows the IAS machinery in and out. Lives and breathes that. H
  - Ramadorai likes him though. Put him on board of Tata Consumers.
- NSE Scam. He’s part of the cabal with Ajay Shah. Private trading firms could co-locate within NSE and could make a huge amount of money. KPK ran some of this by proxy to fund Congress. But he left no fingerprints. But everyone knows it is him. He was running Chitra Ramakrishnan by proxy. He was the Himalayan Yogi.
- Ignite continued with unwavering focus. Kept increasing the kind of focus. We had a 99.5% success rate in placements. Just a handful of failures.
- Ramadorai has written about Ignite in “The TCS Story”. My Dad translated it in Tamil. It’s not a typical business biography. Worth reading. Should be a mandatory course in MBA courses in India. So many lessons.
  - You have to read it knowing how Mr Ramadorai speaks. What is NOT said is just as important. Ch 5 is the thinnest - on the IPO. It is packed with so much stuff. Unless you know, you won’t understand.
  - “Tatas got the Govt to change a tax law to make the IPO meaningful.” Behind that, there’s a lot. You have to be alert to catch the sentene. He won’t brag, or talk about the significance of some of these.
  - Book is packed with dense insights. Unless you ARE LOOKING FOR IT, you’ll miss it. Worth reading SEVERAL times. You need a foot-noting.
- Currently reading Pasquenelli – Social History of Artificial Intelligence. Eye of the Master. Worth reading. I’m not Marxist by belief but they get some things right. Surprised how vibrant the European left is.
  - “If someone is doing manual work, there is tacit knowledge that automation captures.”
- India doesn’t need self-driving cars. But a farmer would like a gaming controller that ploughs his fields while he sits under a tree. Semi-intelligent machines that removes the burden of hard labour in the country.
- Once a year, for a few weeks, I do manual labour. People are under-nourished. People typically work 5 hours a day. Not enough muscle mass. So use them for what they’re good at
- I’ve seen the power tools. When Chinese power tools became cheap, the power welding became much more efficient. Everyone has become a monkey with power tools. They charge per inch. They know how to leverage the tech for economic benefit. Just bring in the power tools and rapidly finish and make money. But there are sections that are still poor and haven’t made the transition. How can we create pathways for them? How can AI help?
- Anand: Why not use a gimball. RS: Good idea.
- Role modern psychologist DW Winnicott on ChatGPT (like Socrates)
  - E.g. You don’t need a perfect mother. A good enough mother is better
  - Similarly, why not a “good enough” Bharat mata than a perfect one?
To persuade someone, align it with their identity. ChatGPT
5 technologies of interest according to Gartner’s latest hype cycle:
- GitOps
- Internal Developer Platform
- Graph Data Science
- Open Source Program Office
- Value Stream Management Platforms
Gemini is an alternative to the Web. Sort of like Gopher, but recent
SALI - Standards Advancement for the Legal Industry - has standards and ontology/taxonomy for legal documents, including patent litigation.
Walking new routes habitualizes fighting fear and preferring novelty
⭐ GPT-4 is bad at math. It gets ~60-70% of answers wrong.
LMQL provides a constraint-based query language for interacting with LLMs. It uses token masking, which is clever.
Hollywood writers signed a deal that limits AI in script writing. It’s primarily aimed at protecting script writer wages.
Adobe Firefly offers a “generative fill” that lets you remove or paint new objects into an image. I’m awaiting text to vector images.
Duet AI is Google’s answer to Github Copilot.
Teachers are using LLMs to plan lessons, write emails to parents, create tests, adjust reading level of materials, personalize content with tools like MagicSchool, Diffit, Eduaide.
WizardLM creates datasets for instruction tuning by cleverly using LLMs to create new prompts.
Deita is an approach to improve instruction tuning datasets.
Dhyeya: Attack on Titan is as good at Death Note
Jaidev:
- Long car drives are a good place to explore new song genres. Try in taxis
- Same radio channels may have different frequencies across cities. Vividh Bharati is 100.5 FM in Chennai and 106.4 in Delhi
- Things to explore:
  - Radio for new songs
  - Clubhouse
  - Twitter Spaces
  - Instagram reels
  - YouTube reaction videos (e.g. atheist, Indian songs, etc.)
  - Stand-up comedies (Ricky Gervais, Louis CK, Jordan Peterson)
Porn artists are at risk because of Gen AI