Teaching my AI assistant from its own mistakes

An AI coding assistant doesn't remember yesterday. If it made a mistake last week and I corrected it, the correction is gone the next time I open a session. Over a few projects I noticed the same kinds of mistakes coming back, and I was spending my time re-explaining things instead of building.

The setup

I use task-observer, an open-source skill for Claude Code written by Eoghan Henn (CC BY 4.0). A global instruction file tells the assistant to load it at the start of every session, check the log for lessons that apply, and, after each task, report what it logged. The point is that noticing a lesson is no longer something I have to remember to do.

Each lesson is one small file with three parts: what happened, what should change, and the general principle. The principle is the field that matters most, and it has to pass a simple test: would it still make sense in a different project? If not, it is a fix for one task, not a lesson.

What five weeks produced

Since 7 September 2026 the log has collected 36 observations. Most of them group around two themes that I didn't plan but that clearly describe how I work: checking that what the code claims is actually true (13 observations), and measuring performance correctly (11). Some of them came straight out of the projects on this site:

• "A measuring instrument must be validated before its numbers are believed, especially when they look consistent." This came from Halcyon's latency benchmark, where three different measuring mistakes each produced a confident, wrong number.
• "Benchmarks pace differently than users do." Also Halcyon: back-to-back requests showed 140 ms, while real conversations took 2.2 seconds because the GPU went to sleep in the pauses.
• "Run the same configuration twice before comparing configurations." Two identical runs of a speech benchmark disagreed by more than ten points, and fixing that reversed an earlier conclusion.
• "Every figure quoted in docs or UI that derives from data should be generated from it." In PerfHub, hand-typed numbers had gone stale in five places at once; the site footer still said 106 measurements and 9.0% error when the real set was 577 at 6.9%.

A lesson that moved to a new project

The last rule is why the PerfHub numbers on this portfolio aren't typed by hand. When the site is built, it reads them from PerfHub's README, where they are regenerated from the data and checked in CI. The lesson came from one project and was applied to another without me having to bring it up.

Building this site added new ones. I wanted the PerfHub section to stay exactly as it was while everything around it changed. Its source file was untouched, but a font or a global style can still change how a component looks. So before the redesign, the computed styles of all 28 elements in that section were recorded, and compared again at the end: zero differences, with a control run showing the comparison could detect one. That became an observation too: "don't touch this part" is checked on what the page renders, not on whether a file changed.

What I haven't done yet

The log is supposed to feed a regular review in which observations become updates to the assistant's skills, or get rejected. I haven't run that review yet, so all 36 are still waiting. Collecting lessons is the easy half; deciding which ones deserve to become rules, and deleting the ones that don't, is the half that actually changes how the assistant behaves. That review is next, and I'll write about what survives it.

What I take from it

Working with an AI assistant is less about writing the perfect prompt and more about building the loop around it: a place where mistakes are written down, a test for whether a lesson is general, and a moment where someone decides what becomes a rule. The assistant writes the observations. Deciding what to do with them is my job.