Local Agents Are Good but Slow

Last year, I shared how local LLMs are expensive. There are only 3 reasons to use them over an inference API:

You have no choice, e.g. Government, pharma, finance, or other sensitive data.
You have free GPUs, e.g. your laptop, local data center with sunk cost, free cloud credits, etc. This doesn’t scale.
You want to learn.

This year, I finally started using them for a combination of #1 (on flights) and #2 (my 8 GB NVIDIA RTX 2000).

It began with simple queries on Edge Gallery on my mobile - like “Is this vegetarian?”, “What do you call those caption boxes in comics?”, etc.

I recently read how easily you can run Pi with Ollama and that ornith:9b and gemma4:e4b-it-qat are good models for my system so I can just run:

ollama launch pi --model gemma4:e4b-it-qat

… to get a reasonable coding model locally.

I tried it for a few non-technical things first. For example:

That morning, I asked my talkative train neighbor, “Are you an extrovert?” He said “No, I’m an introvert.” On the flight, Gemma4 explained the contradiction well
I tried reducing my battery usage on the flight by asking it what different processes were. It didn’t solve the problem but gave me some ideas.

But that’s not to say it’s great. For example, when I asked it:

Can you find a blog post where I spoke about an incident at BCG in the early 2000s (I may have written about it later) where I asked them, “you keep saying we should do more of this and more of that, what should we do less of”? I think there is an entire blog post about that.

Here’s the approach it took (comments are mine):

# List my blog posts - which showed this: https://github.com/sanand0/blog/tree/main/posts
ls ~/code/blog/posts/

# Search for "BCG" - which returned 63 results, NONE of which was the post I needed!
grep -r -i "BCG" ~/code/blog/posts/

# Read the most likely first candidate
cat ~/code/blog/posts/2000/another-day-at-lbs.md

# ...and the next
cat ~/code/blog/posts/2000/boston-consulting-group-interview.md

# .. and the next
path: ~/code/blog/posts/2012/the-next-chapter-of-my-life.md

# Back-track and search for doing more or less in 2000-2003 - again, no results
grep -r -iE "d o more|d o less|sh ould" ~/code/blog/posts/200[0-3]/

# Try across all years - this gives 2718 results
grep -r -iE "more|less" ~/code/blog/posts/

… and it’s been 20 minutes so far - but it probably would have gotten to the answer if it hadn’t crashed for some reason.

On the other hand, GPT-5.6 Sol (on ChatGPT) did the following (comments are mine):

# Write a couple of multi-line scripts to search for "do more", "do less", etc. AS WELL AS for "BCG" and "Boston Consulting" and "consult"
set -euo pipefail
BLOG="$HOME/code/blog/posts"
test -d "$BLOG"
echo '--- phrase matches ---'
rg -n -i --glob '*.md' --glob '*.html' --glob '*.txt' \
  -e 'do more' -e 'do less' -e 'less of' -e 'more of this' -e 'what should we do less' \
  "$BLOG" | head -120 || true

echo '--- BCG/context matches ---'
rg -n -i --glob '*.md' --glob '*.html' --glob '*.txt' \
  -e '\bBCG\b' -e 'Boston Consulting' -e 'consult' \
  "$BLOG" | head -120 || true

# That failed. It provided the wrong home directory: /home/oai/share
#   - It got confused with its own container.
# But it corrected itself and tried again. That got ~220 GOOD matches.

# Now, it does 3 things:
# 1. Read the most promising post (which is the correct one): https://www.s-anand.net/blog/less-is-more/
# 2. ALSO search in the file for related terms like Infosys, manager, office, consultant, years ago, etc.
# 3. A BRILLIANT step - search for other phrases, like "do less... do more" or "what... do less", etc.
#    Just in case it missed something!
set -euo pipefail
F="$HOME/code/blog/posts/2008/less-is-more.md"
echo '--- less-is-more.md ---'
nl -ba "$F" | sed -n '1,125p'

echo '--- nearby explicit provenance clues ---'
rg -n -i -C 3 'BCG|Infosys|manager|office|consultant|years ago|back then|firm' "$F" || true

echo '--- exact/near quote elsewhere ---'
rg -n -i --glob '*.md' -C 2 \
  -e 'you can.t do more of something unless you do less' \
  -e 'what.*do less' \
  -e 'do less.*do more' \
  "$HOME/code/blog/posts" | head -120 || true

That’s it. That’s all it needed to find the post.

What impressed me about both agents is that:

They used smart synonyms just from my query - they know how people typically mis-spell.
They corrected themselves as they went along. For example:
- Pi found no results between 2000-2003, and extended to all my blog posts.
- ChatGPT got the home directory wrong, but corrected itself.
They revised strategies as they learnt. For example:
- Pi realized I might not have mentioned “BCG” in my post.
- ChatGPT realized I might have multiple blog posts about this.
- Both updated search terms to use based on the results. For example, ChatGPT learnt I was at Infosys from the search results and added that as a search term!

So, I’ve reached a mental tipping point. Local agents are good enough in an emergency. They’re slow, but they can my work done if needed.

Related