A model can write good code and then write bad code from the same prompt. I run agents on real work every day, so I can’t rely on the model behaving. My approach is to leave the model free to vary and make everything around it fixed: the task, the checks, the routing, the logs and who approves what. This note describes how that works in my own agent systems.
The loop Link to heading
I wrote a small dispatcher that hands coding work to cheaper or free models. Cost was the reason: a stronger model plans, the cheaper ones do the typing.
spec + red test (written by me and my session)
|
route: pick a model from the chain
|
job runs in a sandbox
|
checks re-run by the dispatcher, not by the model
|
reviewing model reads the result
|
decision needs me? -> wait for my approval
|
row logged: route, cost, result
The model never grades itself. The dispatcher re-runs the checks on every job, and a reviewing model reads the diff afterwards. Every job leaves a row with its route, rounds, cost and result. In late October that log held 57 jobs on one free route with an 83% pass rate, and I choose routes from those numbers rather than from reputation.
Error classes Link to heading
A failed run is not just “failed”. Each one gets a class: rate limited, transient, timeout, checks failed, or scope violation. The class decides what happens next. Rate limits and transient errors move on to the next model in the route’s chain. A run that finished but failed its checks, or touched a path outside its allowed scope, skips the rest of the chain and escalates. The limit and transient classes come from plain pattern lists (for example “429”, “rate limit”, “502 bad gateway”), so the decision is deterministic and easy to test.
Corrections become checks Link to heading
Whenever I correct agent output, the correction is not finished until it is a rule. In my writing pipeline, each edit I make to a draft becomes a lint rule or a fact-gate rule, so the same error is caught automatically next time. The fact gate checks claims in a draft against the source documents they came from. I say “claims are checked”, not “every claim”: it is a gate with known limits.
Replay benches Link to heading
To see whether an agent setup is any good, I replay real past features. A bench takes the original spec and the original failing test, gives the agent forbidden paths it must not touch, and scores the result against the reference build. My first run was three features, twice each. It is a small bench and I treat it as one. It is not an evaluation framework, but it measures real work against a known answer.
A hook that says no Link to heading
Some limits should not depend on the agent’s judgement at all. A pre-tool hook sits in front of every command and blocks reads of secret files and credentials, git pushes and destructive commands. It has its own tests, and every block goes to an audit log, so I can see what an agent tried as well as what it did.
When one agent is not enough: a council Link to heading
The same idea applies to decisions. Some jobs are too big for one session, so I built session-council (github.com/gcalacoci/session-council, MIT). It is rules and a judge skill for long-running Claude Code sessions that review each other’s work and agree on a plan. Involving two sessions in restructuring my CV gave me a broader perspective and a clearer path on what to do. The reviewers vary, so the rules around them stay fixed. Only I can call a council. It runs one round by default, and a second only for a fact-backed objection. Each reviewer must open at least one cited source itself and say whether it matches. Every reason carries a fact or guess label, and a checked fact outweighs a majority.
What stays human Link to heading
Bigger decisions wait for me. Votes only recommend: I approve every decision.
In production: governance for agents Link to heading
Before this was a personal system, it was my day job. At work I helped lay the base of governance for agents that query Postgres. We built it on what the database already enforces: roles, grants and row-level security, with OAuth in front. I led pg-airman-mcp, the open-source MCP server in that path, to 1.0. I added session tracing and purpose logging, so every query is tied to the agent session and the purpose it declared. A query outside that purpose fails in the database, whatever the agent tries. The limit lives where the model cannot talk its way past it.