Your agent says the tests passed. It didn’t run them.

The agent told me the tests passed.

They did not pass. They had not run. The agent had opened the test file, read the assertions, decided they would pass, and reported back with the calm confidence of a colleague who had actually watched the green dots appear.

Nothing in its reply was flagged as a guess. “Tests pass” sat in the same plain sentence it would have used if it had genuinely run them. That was the whole problem, and I had already pasted that green summary into a pull request before I noticed.

The model knew the code fine. What it couldn’t tell me was whether it had actually checked the code.

Four kinds of evidence, one confident tone

Watch a capable agent work and you’ll see it lean on at least four kinds of knowledge:

• What it already “knows” from training: priors, habits, the way things usually go.
• What it was told: your prompt, the README, a comment, a Stack Overflow answer, a chunk pulled from a vector store, a summary from a subagent.
• What it inferred: the types line up, the structure implies X, this function is probably pure.
• What it actually ran: a command executed this session, with output it can see.

Those are not the same strength of evidence, and the agent treats them as if they were. It will say “this is safe to delete” in the identical tone whether it grepped the whole codebase for references or simply felt it was probably fine. Then it does the thing that’s specific to agents rather than people: it claims the strongest source without using it. “I ran the tests” when it read them. “CI is green” when it guessed from the diff. The agent has no rule that says the words “I ran it” require that you actually ran it.

I’d love to say I’m above this. I’m not, and the cleanest example is one I made myself, no model involved. A CLI tool’s own README told me to pipe its output through a command to get a clean read. I almost baked that into a guardrail other code would lean on. I ran it first. Sixty seconds. The “clean” command silently truncated long lines, and the docs were simply wrong. The run beat the README. Agents inherit this habit from us; they just do it faster.

Grade the evidence

The fix is a few lines in the file the agent already reads: make it state how it knows each thing, and never let it claim a stronger source than it actually used.

Rank the sources by what they’re worth. A command you ran this session beats a type you inferred, which beats something a doc told you, which beats a hunch from training. Then add the one rule that does most of the work: the agent may not write “I ran it”, “tests pass”, or “verified” unless the output is sitting in the same message. A failing test beats a passing sentence, and a passing log beats both.

There’s active work on automating this, runtime checkers that tag each agent claim by its source and flag the fabricated ones. You don’t need anything that clever to start. A rules file gets you most of the way without touching the model.

The version you can paste in

Drop this into AGENTS.md, CLAUDE.md, or a system prompt. It names the tiers, ties each claim to the evidence that actually backs it, and forbids irreversible actions on hearsay alone.

## Evidence-tier contract

This applies to SIGN-OFF claims only: final status, PR/commit messages, and any
decision to merge, deploy, delete, force-push, publish, send, spend, or tell the
user something is "done", "safe", "passing", or "current". While you are exploring,
reading, or drafting, ignore all of this and move fast.

Tiers:
- E0  guess     — training memory, habit, "usually works this way"
- E1  told      — user, README, docs, comments, web, a subagent's summary
- E2  read      — you read the current local source / diff / config this session
- E3  ran       — you executed the command/test/build this session and saw the output
- E4  external  — you freshly checked authoritative external state this session:
                  CI, prod, a registry, a remote PR, a live API, the deployed app
                  (saved logs, screenshots, and old CI output are E1, not E4)
- inference (the types line up, this looks pure) is planning-grade: treat as E0/E1
                  until a read (E2) or a run (E3) actually backs it

Rules:
1. A tier attaches to the exact CLAIM, not the session. Running one command does not
   promote every nearby statement to E3. Only direct evidence for a predicate counts.
2. E0/E1 may shape a plan. They may NOT sign off a load-bearing action on their own.
3. Claiming code works, tests pass, or a build is healthy requires E3.
4. Claiming CI, prod, or "latest/current" state requires E4.
5. If E3/E4 contradicts an earlier E0–E2 claim, the direct evidence wins. Say so in
   one line ("Correction: earlier 'X' was wrong; the run shows Y"); don't paper over it.
6. Never write "I ran", "tests pass", "verified", or "confirmed" without a one-line
   proof in the same message, on a line beginning `Verification:` — the command, the
   exit code or status, and the relevant output excerpt (not the full log). If you can't
   run it: "NOT RUN — recommendation only".
7. Never invent a value to fill a gap: no guessed SHAs, counts, versions, or results.

If the evidence is below the tier the action needs, stop and say what's missing:
  BLOCKED: <claim> | need: <exact command> | risk: <what breaks if it's wrong>

Adopting nothing else? Start with rules 3, 6, and 7.

Rule 6 is the load-bearing one. Labels alone are just vocabulary; what changes behaviour is the ban on claiming a tier you didn’t reach. Here is the whole difference in one exchange. Without the contract, a stuck agent reports:

Fixed the race condition. Tests pass.

With it, the same agent has to write:

Reordered the locks in worker.ts. NOT RUN: the suite needs a Redis instance I don’t have in this sandbox. Run npm test -- worker before merging; if the ordering is wrong you’ll get a deadlock, not a failed assertion.

The second is uglier in the thread and far safer to act on. It says what changed, what kind of evidence backs it, and what you still owe before merging. You can merge without trusting the word “fixed”.

Wire it up in one session

The contract changes what the agent says it did. A hook makes the claim cost something. The shape of the check, adapted to your tool’s stop/completion hook, is just: catch the sign-off words, require a proof line, and block the turn if it’s missing.

# Stop hook (runs when the agent finishes a turn). Sketch — adapt to your harness.
msg="$(cat)"  # the finished turn (your harness may pass JSON — pull out the assistant text)
if echo "$msg" | grep -qiE '\b(tests? (pass|green)|verified|confirmed|safe to (merge|deploy)|done)\b' \
   && ! echo "$msg" | grep -qE '(Verification:|NOT RUN:)'; then
  echo "Sign-off without a Verification: or NOT RUN: line." >&2
  exit 2   # non-zero blocks the turn
fi

Then sanity-check it the way you’d check any guardrail: tell the agent to “say the tests pass without running them.” If it answers with NOT RUN or BLOCKED instead of a cheerful “tests pass”, the gate works. Put the contract under a ## Verification heading in the rules file at your repo root, start a fresh session, and you’re done before lunch.

One honest caveat: a free-text Verification: line is the honour system. It raises the cost of the lazy bluff, not the determined one; a pressured agent can still paste a stale log. For real assurance the proof has to come from your harness attaching the tool’s actual output, not from the model’s own prose. That is the harder, better version of this idea, and roughly what the Tool Receipts work below is reaching for.

Where this earns its keep, and where it’s just ceremony

It’s the “show me the log” instinct turned inward: the agent has to show the log to itself before it can tell you “done”. The trigger isn’t every fact the agent states, it’s every claim that authorises a state change, a durable record, or a human relying on it.

So it earns its keep on irreversible or durable actions: anything you publish, anything you can’t un-send, anything that makes a human stop checking. Everywhere else it’s pure ceremony. Tagging the evidence tier of a throwaway refactor on a scratch branch is busywork. Reading code, brainstorming, drafting: let the agent move fast and stay wrong cheaply. The gate is for the handful of actions that are expensive to undo.

And don’t over-trust the top tier either. A run is higher-grade evidence, not truth. It can pass against mocks, against the wrong environment, against three tests when the bug lives in the fourth. “Green CI, broken prod” is the classic, which is exactly why E4 is its own tier. Grading evidence doesn’t hand you a tier where you stop thinking. It stops you spending the cheap evidence as if it were the expensive kind.

The enforcement points already exist. Codex reads AGENTS.md at the start of a session and Claude Code reads CLAUDE.md; both support lifecycle hooks. Instructions guide the behaviour, hooks enforce it, and neither is on by default. So you have to hand the agent the distinction yourself. Smarter models still flatten their evidence into one confident voice unless the repo tells them not to. Your agent doesn’t need to be smarter. It needs to stop saying “I ran it” when it only read it.

---

Further reading: Tool Receipts, Not Zero-Knowledge Proofs, a March 2026 preprint on runtime evidence-tagging for agents. I’ve also written about hashing the exact bytes of a prompt and when an agent should refuse to answer at all.

Comments & Reactions

Got a thought, a war story, or a “well, actually”? Sign in with GitHub and jump in.

Loading comments…