testing on S Anand

Hardening my Dev Container Setup

Sat, 14 Mar 2026 12:30:35 +0530

I run AI coding agents inside a Docker container for safety.

The setup is

dev.dockerfile: builds the image
dev.sh: launches the container with the right mounts and env vars
dev.test.sh: verifies everything works.

I wrote them semi-manually and it had bugs. I had GitHub Copilot + GPT-5.4 High update tests and actually run the commands to verify the setup.

Here’s what I learned from the process.

1. Make it easier to review. The first run took long. I pressed Ctrl+C, told Copilot to “add colored output, timing, and a live status line”. Then I re-ran. Instead of a bunch of ERROR: lines, I now got a color-coded output with timing + a live status line showing what’s running.

PASS   2.495s  markitdown --help
FAIL   0.003s  playwright --version
note /home/sanand/code/scripts/dev.test.sh: line 148: playwright: command not found
FAIL   0.003s  copilot version
note /home/sanand/code/scripts/dev.test.sh: line 148: copilot: command not found

2. Run in the target environment. Tests reported that fd and node were missing, though they were obviously in the image. Why? The test script was running on my laptop, not in the container. It was checking the host, not the container. I didn’t realize that.

Copilot added a check to see if we’re already inside a container:

running_in_container() {
  [ -f /.dockerenv ] || grep -qaE '(docker|containerd|kubepods)' /proc/1/cgroup 2>/dev/null
}

if ! running_in_container; then
  exec bash "${SCRIPT_DIR}/dev.sh" -- "$SCRIPT_PATH" --inside-container
fi

3. Check for conflicting commands. fd and node were still missing. Why? dev.sh was mounting my host mise and bin directories, overshadowing the image

-v "$HOME/.local/share/mise:/home/vscode/.local/share/mise"
-v "$HOME/.local/bin:/home/vscode/.local/bin"

It doesn’t matter what’s installed in the container - only the host binaries are visible. Copilot removed the first line, and copied mise into ~/.local/overrides/ to override the ~/.local/bin PATH.

4. Use the right packages. The default ImageMagick binary at https://imagemagick.org/archive/binaries/magick is an AppImage. It doesn’t work in containers. So it hat to switch to apt-get install -y imagemagick instead.

5. Use the right paths. npm install -g wscat playwright ... ran but to add them to the PATH, you need to mise reshum node. Copilot added that.

These are good practices, but since agents can fix them, they’re less important to learn.

Vibe-coding good tests enabled Copilot to fix them.
Making output easy to read enabled me to steer Copilot.

That’s what I’d recommend if you’re trying to optimize setup / deployments.

Mistakes AI Coding Agents Make

Sun, 22 Jun 2025 09:21:05 +0000

I use Codex to write tools while I walk. Here are merged PRs:

This added technical debt. I spent four hours fixing the AI generated tests and code.

What mistakes did it make?

Inconsistency. It flips between execCommand("copy") and clipboard.writeText(). It wavers on timeouts (50 ms vs 100 ms). It doesn’t always run/fix test cases.
Missed edge cases. I switched <div> to <form>. My earlier code didn’t have a type="button", so clicks reloaded the page. It missed that. It also left scripts as plain <script> instead of <script type="module"> which was required.
Limited experimentation. My failed with a HTTP 404 because the common/ directory wasn’t served. I added console.logs to find this. Also, happy-dom won’t handle multiple exports instead of a single export { ... }. I wrote code to verify this. Coding agents didn’t run such experiments.

What can we do about it?

Three things could have helped me:

Detailed coding rules. E.g. always run test cases and fix until they pass. Only use ESM. Always import from CDN via JSDelivr. That sort of thing.
100% test coverage. Ideally 100% of code and all usage scenarios.
Log everything. My tests got a HTTP 404 because I was not serving the common/ directory. LLMs couldn’t figure this out because it was not logged. Logging everything helps humans and LLMs debug.
Wait. LLMs and coding agents keep improving. A few months down the line, they’ll run more experiments themselves.

Was AI coding worth the effort? Here, yes. The tools worked. Codex saved me 90% effort. My code quality obsession reduced savings to ~70%. Still huge.

Turning Walks into Pull Requests

Fri, 30 May 2025 05:16:52 +0000

In the last few days, I’m coding with Jules (Google’s coding agent) while walking.

Here are a few pull requests merged so far:

Why bother? My commute used to be audiobook time. Great for ideas, useless for deliverables. With ChatGPT, Gemini, Claude.ai, etc. I was able to have them write code, but I still needed to run, test, and deploy. Jules (and tools like GitHub Copilot Coding Agent, OpenAI Codex, PR Agent, etc. which are not currently free for everyone) lets you chat clone a repo, write code in a new branch, test it, and push. I can deploy that with a click.

Fifteen minutes into yesterday’s walk I realised I’d shipped more code than in an hour at my desk (even with LLMs)!

Workflow

Open Jules via browser on phone, connect wired headset.
Prompt (by typing or speaking) the change to make. It reads the repo, creates a plan and writes code.
It runs any existing test suites in a sandbox. Repeats until all tests pass.
I have it publish a branch, go to GitHub Mobile and create a PR.
Back home, I review the output and merge.

There are 3 kinds of uses I’ve put it to.

#1. Documentation is the easiest. Low risk, high quality, boring task. Here’s a sample prompt:

This repo has multiple directories, each with their own standalone single page application tools.

If a directory does not have a README.md, add a concise, clear, USEFUL, tersely worded one covering what the tool does, the various real life use cases, and how it works.

If a readme already exists, do NOT delete any information. Prefix this new information at the start.

Avoid repeating information across multiple README files. Consolidated such information into the root directory readme.

In the root directory README, also include links to each tool directory as a list, explaining in a single sentence what the tool does.

#2. Testing is the next best. Low risk, medium quality, boring task. Here’s an example:

Run the tests in this repo. Go through the code and see what parts of the code are not covered. Understand the logic and see what kinds of user scenarios are not covered. Add test cases to cover these in the same style as the existing code.

Write MINIMAL, ELEGANT code.

#3. Coding may not be the best suited for this. High risk, medium quality, and interesting. But here’s a sample prompt:

Fix https://github.com/sanand0/tools/issues/3

Allow the user to enter just the GitHub @username, e.g. @sanand0 apart from the URL

Add crisp documentation at the start explaining what the app does

Only display html_url (as a link), avatar_url (as an image), name, company, blog, location, email, hireable, bio, twitter_username, public_repos, public_gists, followers, following, created_at, updated_at

Format dates like Wed 28 May 2025. Format numbers with commas. Add links to blog, twitter_username, email

Add “Download CSV” and “Copy to Excel” buttons similar to the json2csv/ tool

Automated tests are a great way to reduce AI coding risk, as Simon Willison suggests. I need to do more of this!

Wins & Losses

Good: 1 walk = one merged PR. Even with LLMs, it used to take me 2 hours. Now, it’s about half an hour of reclaimed walking “dead time”.
Good: Test-first prompting caught a sneaky race condition I’d have missed.
Bad: Told Jules “add docs” without saying “don’t overwrite existing.” It politely destroyed my README. Manual revert ensued.
Bad: Front-end tasks need visual QA; I’m still hunting for a zero-setup UAT preview on mobile.

The industry echoes the pattern: GitHub’s new Copilot agent submits draft PRs behind branch protections [1]; Sweep auto-fixes small tickets but can over-touch files [2]; Microsoft’s own engineers found agents flailed on complex bug fixes [3].

But…

Isn’t this risky? Maybe. Branch protections, CI, and human review stay intact. Agents are like a noisy junior devs who never sleep.
Is the diff readable? If not, I have it retry, write more reviewable diffs, and explain clearly in comments & commit messages.
Does it have enough context? I add all the context clearly in the issue or the prompt. That can take some research.
Security? The agents run inside repos you give it access. Prompt injection and exfiltration are possible risks, but only if it accesses external code / websites.

How to start

Pick a low-stakes repo with solid tests.
Pick an agent. Jules has 5 tasks/day free for now. Or pay and use GitHub Copilot Coding Agent, OpenAI Codex, etc. Or self-host PR Agent, etc.
Write a failing test.
Go for a walk and talk.
Merge (or laugh) on return.

Leaning into the power of AI coding

Wed, 16 Oct 2024 03:55:59 +0000

Yesterday (15 Oct 2024), I used Cursor to code more than I ever have. (Doing's how we learn, I guess. Not just reading.)

This was mainly to create and publish 2 libraries on npm over 6 hours:

asyncsse - which converts a Server-Sent Event stream into an async iterator that I can use in a for await ... of loop
asyncllm - which standardizes the Server-Sent Events streamed by the popular LLMs into an easy to use form.

This exercise broke several mental barriers for me.

Writing in a new language. Deno 2.0 was released recently. I was impressed by the compatibility with npm packages. Plus, it's a single EXE download that includes a linter, tester, formatter, etc. Like all recent cool fast tools, it's written in Rust. So I decided to use it for testing. Running deno test runs the entire test suite. My prompts included asking it to:

Create a Deno HTTP server to mock requests for the tests. This is cool because a single, simple code chunk runs the server within the test suite.
Serve static files from samples/ to move my tests into files

Writing test cases. Every line of this code was written by Cursor via Claude 3.5 Sonnet. Every line. My prompt was, Look at the code in @index.js and write test cases for scenarios not yet covered. It's surprising how much of the SSE spec it already knew, and anticipated edge cases like:

SSE values might have a colon. I learnt for the first time that the limit parameter in String.split() is very different from Python's str.split. (The splits, then picks the first few, ignoring the rest. Python ensures the rest is packed into the last split.) This helped me find a major bug.
SSE has comments. Empty keys are treated as strings. Didn't know this.

I was able to use it to generate test cases based on content as well. Based on @index.js and @openai.txt write a test case that verifies the functionality created the entire test case for OpenAI responses. (I did have to edit it because LLMs don't count very well, but it was minimal.)

Bridging test coverage gaps. The prompt that gave me the most delightful result was Are there any scenarios in @index.js not tested by @test.js? It did a great job of highlighting that I hadn't covered Groq, Azure, or CloudFlare AI workers (though they were mentioned in the comments), error handling, empty/null values in some cases, tested for multiple tool calls. I had it generate mock test data for some of these and added the tests.

Enhancing knowledge with references. I passed Cursor the SSE documentation via @https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events and asked it to find more scenarios my code at @index.js had not covered. This found a number of new issues.

Generating bindings. I avoid TypeScript because I don't know it. Plus, it requires an compilation step for the browser. But TypeScript bindings are helpful. So I prompted Cursor, using the Composer (which can create new files) to Create TypeScript bindings for @index.js in index.d.ts - which id did almost perfectly.

Check for errors. I typed Check this file for errors on @index.d.ts. I don't know enough to figure this out. It went through the description and said everything seems fine. But I saw a TypeScript plugin error that said, Property 'data' of type 'string | undefined' is not assignable to 'string' index type 'string'.ts(2411). When prompted, it spotted the issue. (The earlier code assumed all properties are strings. But some can be undefined too. It fixed it.)

Documentation. At first, I asked the Composer to Create a README.md suitable for a world-class professional open source npm package and it did a pretty good job. I just needed to update the repository name. I further prompted it to Modify README based on @index.js and share examples from @test.js on asyncllm, which did an excellent job.

Code review. I asked it to Review this code. Suggest possible improvements for simplicity, future-proofing, robustness, and efficiency and it shared a few very effective improvements.

Regex lookaheads for efficient regular expression splitting, i.e. use buffer.split(/(?=\r?\n\r?\n)/) instead of buffer.split(/(\r?\n\r?\n)/) -- and though I haven't tested this, it looked cool.
Restructuring complex if-else code into elegant parsers that made my code a lot more modular.
Error handling. It added try {} catch {} blocks at a few places that helped catch errors that I don't anticipate but don't hurt.

Code simplification. Several times, I passed it a code snippet, saying just Simplify. Here's an example:

const events = [];
for await (const event of asyncLLM(...)) {
  events.push(event);
}

This can be simplified to

const events = await Array.fromAsync(asyncLLM(...))

Packaging. I copied a package.json from an earlier file and asked it to Modify package.json, notable keywords and files and scripts based on @index.js which it did a perfect job of.

Blogging. I wrote this blog post with the help of the chat history on Cursor. Normally, such blog posts take me 3-4 hours. This one took 45 minutes. I just had to pick and choose from history. (I lost a few because I renamed directories. I'll be careful not to do that going forward.)

Overall, it was a day of great learning. Not in the classroom sense of "Here's something I didn't know before", but rather the cycling / swimming sense of "Here's something I now know to do."