How I build software with an AI coding assistant

Most of the code in Halcyon and PerfHub was written together with Claude Code, an AI coding assistant. You can see it in the commit history: those commits carry a "Co-Authored-By" line. I would rather say that plainly than leave it for someone to discover, so this post is about what that collaboration looks like in practice.

The short version: the assistant makes it cheap to produce code, tests and documentation. That moves the hard part somewhere else, to deciding what "correct" means and proving that what was produced actually is. That part is mine.

I define what "done" means before anything is written

PerfHub estimates frame rates for PC hardware. A model like that can always print a number; the question is how wrong the number is. So the definition of done was never "the page shows FPS". It was "every figure we publish is measured against recorded benchmarks, including on games the model has never seen". The README figures are regenerated from the data by a script, and CI fails if one of them goes stale. Games that were never measured are labelled as estimates, with their tested error next to them.

Plausible output is not evidence

An early Halcyon benchmark reported almost exactly 3.7 seconds on every input. The numbers looked consistent, which is what made them suspicious. It turned out the benchmark abandoned each response stream after the first sentence while the server kept generating, so every measurement queued behind the previous one. The tool was measuring itself.

Suspiciously uniform numbers were the clue. A result that looks too clean deserves a second instrument.

Test the way people actually use it

Another benchmark said the language model answered in 140 ms. In real use, the first answer took 2.2 seconds. Both were right. The benchmark sent requests back to back; a person pauses to speak and listen, and in that gap the laptop GPU drops into a low-power state that takes about two seconds to leave. The fix was to wake the GPU the moment you start speaking, so the wake-up overlaps with time that is already being spent.

Make a measurement repeatable before comparing anything

Two runs of Halcyon's speech-recognition benchmark with identical settings disagreed by more than ten points of error rate. There were three hidden sources of randomness: the speech synthesiser that generated the test audio, Whisper's retries on noisy audio, and background noise that depended on the order clips were generated in. With all three pinned, two runs now produce identical tables, and one earlier conclusion reversed.

Look for things that work by accident

GPU speech recognition in Halcyon worked only because PyTorch happened to be imported first and had already loaded an NVIDIA library that faster-whisper needed. Measuring Whisper on its own crashed. The dependency is now explicit instead of depending on import order. Code that works for a reason nobody wrote down will stop working for a reason nobody can find.

What I keep for myself

• Choosing the problem and deciding what "working" means.
• Reviewing every change before it is committed.
• Running the measurements myself. A number goes into a README only after I have reproduced it.
• Fundamentals: algorithms, data structures, OOP and SQL, which I practise by hand in coursework and in small Java projects.

What comes next

Next I want to experiment with several agents working together on one task. I expect the same rule to hold, only more so: the more output gets produced automatically, the more each claim needs a check that does not come from the same place as the claim.