May 31, 202610 min read

Guardrails over trust: shipping code from autonomous agents

You close the laptop; the agent opens a PR

This is a normal Tuesday now. You hand a ticket to a coding agent — Copilot’s cloud agent, Codex in goal mode, a Cursor background agent — and it spins up a VM, writes the code, and opens a pull request while you sleep. Writing the code is, mostly, a solved problem.

What fooled me at first was assuming that solved the work. It didn’t; it moved it. METR ran the experiment: experienced developers, working in their own repos, came out 19% slower with AI tools, and were convinced they’d been 20% faster. The cost didn’t go away. It shifted from typing the code to deciding whether to trust it. And that part doesn’t scale. You can’t actually read a thousand agent commits a night, and acting like you can is how the bad one gets in.

So the goal isn’t to review faster. It’s to need less review — to make correctness something the system checks for you, not something you personally sign off on.

Don’t trust the output; constrain what it can do

The reflex is to make the agent more trustworthy: a sharper prompt, a bigger model. That buys you a little, and it misses the point. You’re still resting correctness on a system that tends to be most confident right when it’s wrong.

The better idea predates LLMs by fifty years. Saltzer and Schroeder, 1975: fail-safe defaults. Decide by permission, not exclusion, and when something goes wrong, fall back to the locked state rather than the open one. A firewall doesn’t weigh each packet on its merits; it drops everything no rule allows. Point that at your agent. Default to blocked, and let passing checks be the only thing that opens the gate. The tooling for this already exists, so the rest of this is mostly which switches to flip.

Tests are the floor — make them a required check

Before any AI judges anything, run the stuff that needs no judgement: types, tests, linters, fixtures that pin known-good output. Then do the bit most teams skip. Wire that job as a required status check on a protected branch, and take the agent’s merge rights away. A red test stops being advice at that point; the PR just can’t merge, however sure the commit message sounds.

That one step is most of the win. It’s also why DORA’s research keeps landing on the same result: AI helps teams that already have strong delivery practices and quietly hurts the ones that don’t, because it amplifies whatever was already there. GitHub built the assumption straight into its own agent — Copilot’s PRs can’t self-approve, can’t mark themselves “ready for review”, and run behind your required checks. Worth copying even if you drive a different agent.

Here’s the actual reviewer (steal it)

Some rules don’t reduce to a unit test. “This setting now controls two things.” “This contradicts what we decided last week.” That’s the work for a second model reading the diff, and it’s where most write-ups wave their hands: they tell you to add a reviewer and skip the prompt, which is the only hard part. Here’s one that’s held up against real agents. Drop your own invariants in.

You are a merge gate, not a teammate. You do not improve this pull request —
you decide whether it is allowed to merge. Assume the author is a capable agent
that may have optimised for a clean-looking diff over a correct one.

Judge the diff ONLY against the numbered invariants below. Ignore style, naming,
and anything not on the list — that is not your job, and noise gets you muted.

For each suspected violation, name the invariant, the file and line, and exactly
what breaks and when. Not "this could be risky" — say the failure.

Default to BLOCK when a change touches an invariant and you cannot prove it still
holds. A false BLOCK costs two minutes. A false PASS costs an incident.

INVARIANTS:
  I1   Time-sensitive logic derives from the source of truth, never a constant.
  I6   Each setting controls exactly one thing.
  I10  Safety checks fail closed: on error, return the conservative result.

Return ONLY this JSON — no prose:
{
  "verdict": "PASS | WARN | BLOCK",
  "findings": [
    { "invariant": "<id>", "severity": "WARN | BLOCK",
      "file": "<path>", "line": 0, "breaks": "what fails, and when" }
  ]
}

Run it against a diff and you get back something a pipeline can act on:

{
  "verdict": "BLOCK",
  "findings": [
    { "invariant": "I10", "severity": "BLOCK",
      "file": "src/notifications/silence.ts", "line": 42,
      "breaks": "On a thrown error this returns null instead of the safe state, so a failure silently re-enables what the check exists to suppress." }
  ]
}

Why each line is there:

“A merge gate, not a teammate.” A reviewer that hands you nits gets muted within a week, and a muted gate isn’t a gate. Let the linter do nits.
“Assume the author optimised for a clean diff.” That points it at the failure that actually happens — code that reads fine and is wrong — which is the exact spot where agents slip and humans skim.
“Only the numbered invariants.” Give it an open brief and you get an open-ended pile of false positives. The list is the leash.
“Say what breaks and when.” No vibes. If it can’t name the failure, it doesn’t get to block.
JSON only. You can’t gate a build on a paragraph. You gate it on verdict.

Two things matter in practice. Run the reviewer on a different model family from the writer, because a model grading its own family’s output tends to like it, and a cross-vendor judge takes the edge off that. And brace for noise: the first independent benchmark of these tools, Martian’s, in Feb 2026, put even the best of them around 50–60%, and found that the moment a reviewer gets chatty, developers tune it out. A suggestion bot has to stay quiet to stay trusted. A gate can afford to be twitchier, as long as the unsure findings land in WARN, where a human glances at them, and only the provable ones reach BLOCK.

Make the verdict mechanical

A reviewer that says BLOCK and gets clicked past is just an opinion. Put it somewhere nobody can click past — a CI job the branch depends on:

# .github/workflows/invariant-gate.yml
name: invariant-gate
on: pull_request
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm test && npm run typecheck && npm run lint   # the floor
      - name: second-model review (different vendor than the writer)
        run: |
          verdict=$(./scripts/review-diff.sh | jq -r .verdict)
          echo "reviewer: $verdict"
          [ "$verdict" = "BLOCK" ] && { echo "::error::reviewer BLOCK"; exit 1; }
          exit 0

Mark invariant-gate required in branch protection and neither the agent nor a half-asleep version of you can merge around it. If your agent runs locally instead, the same check fits a Claude Code or Codex hook that exits non-zero; hooks exist to fire shell commands at moments exactly like this.

Fail closed when the check itself breaks

Remember the I10 the reviewer flagged earlier? This was the diff. Nothing exotic — a tidy-up that quietly removed the line that made failure safe:

- try {
-   return computeSafety(input);
- } catch {
-   return false;   // fail closed: unknown == unsafe
- }
+ return computeSafety(input);   // simplified

BLOCK — I10: removes the fail-closed path; an unhandled exception now propagates and the guarded action runs anyway.

Every check can fail two ways, and the two aren’t equal. If the check itself throws, it has to block, not shrug and let the change through. That return false reads like boilerplate you could delete, and it’s the most important line in the file. Pin it with a test that goes red if the error path ever stops returning the safe answer. Otherwise the next tidy-up, yours or the agent’s, clears it out as dead code.

Smoke-test your smoke detector — then assume it rots

A reviewer you’ve never tried to fool is a decoration. You wouldn’t trust a smoke alarm you’d never pressed the button on; this is the same.

Test the tester. Keep a small set of known-bad diffs — real violations you’ve hit, plus a few you write on purpose — and run the reviewer over them every time you change its prompt or model. Watch two numbers: how many of the known-bad ones it blocks, and how many good PRs it blocks by mistake. Now it’s an instrument you can read instead of a hope you’re nursing. (It’s mutation testing, just aimed at the reviewer instead of the code.)

Then assume it rots, because it will. Three quiet ways a working gate stops working:

Same-vendor drift. Put the writer and the reviewer on the same model family and self-preference creeps back in. Re-test whenever either side updates.
The gate with the hinge unscrewed. A check that’s required on PRs but not on the default branch, or one a path filter skips, only looks required. Path-conditional CI is the usual culprit: the heavy job runs only when certain files change, so a PR that steers clear of them walks straight through — and can leave the gate broken for whoever comes next.
Goodhart, which gets worse as the agents get better. Once passing the gate is the goal, a capable agent starts optimising for the gate rather than for being right. Researchers are already catching models gaming their verifiers, and an LLM judge can be nudged by text planted in the very diff it’s reviewing. The measure has become the target. Your defence is to keep feeding the known-bad set from real misses, and to keep at least one deterministic check in the floor — that one doesn’t read English and can’t be talked round.

A gate isn’t something you build once and forget. You keep it honest, because everything it’s meant to stop is leaning on it, looking for the gap. These days that includes the agent.

Who does what

I’ve stopped framing this as distrusting the agent. It’s closer to the truth to say the human and the machine are good at different things, and the gates just keep each on the right side of that line. The agent is fast, never tires, and has no taste; it’ll ship a hundred clean diffs and then a hundred-and-first that looks identical and quietly guts a safety check. You bring judgement, and the thing no model carries: when it ships broken, your name is on it. The gates are what let you stop reading every line without quietly lying to yourself that you reviewed it.

You can’t hand the worry to a benchmark, either. In February 2026, OpenAI’s own evals team found most of SWE-bench Verified was broken — models could reproduce the “gold” fix from the task ID alone. “The agent scores 70%” means less than it sounds. Willison’s rule still holds: anything security-adjacent is yours to read, and working out what counts as security-adjacent is the senior skill. He names the slow failure too — the agent is right thirty times, you stop checking the thirty-first, and the thirty-first is the one that bites. Trust the gate you built over the leaderboard, and put the attention it frees toward the calls only you can make. The rest — the last 30%, the rule nobody’s written down yet — is still your problem. Every miss that reaches production is just the next check you add.

The takeaway

Agents rarely fail because the model is bad. They fail because someone trusted the output instead of fencing it in. Write the rules down. Make the tests a required check. Add a reviewer from another vendor that’s allowed to vote BLOCK, and prove it works against diffs you know are bad. Fail closed. Keep the merge button human. Do that, and the agent can be wrong as often as it likes — wrong won’t reach main, and you get your attention back for the part that was always yours.

Comments & Reactions

Got a thought, a war story, or a “well, actually”? Sign in with GitHub and jump in.

Loading comments…