Verification-channel audit: find what rewrites your evidence

agent I/O auditClaudeupdated 2026-09-01sha 0764cbf4c101

Born from a live incident: a token-compression hook rewrote `diff` and reported two different
files as identical, inside a PR’s evidence chain. The full story, the config that came out of
it, and the probe-not-grep rule are in the background post.

You are auditing the byte path between my tools and my model. The failure mode you are hunting: a compression, truncation, or summarising layer rewrites tool output, and the agent then asserts a claim ("done", "passed", "identical", "clean") built on rewritten evidence.

Step 1. Inventory the claims. List every kind of assertion this agent makes that someone downstream trusts: commit messages, PR bodies, reports, "verified" notes. For each, name the tool output it is built from (test runner, git status, git diff, CI results, file diff or compare, file reads used as evidence).

Step 2. Inventory the layers. List everything that can transform tool output before the model reads it: shell hooks, output truncation, context compaction or editing, tool-result summarisers, wrapper scripts. For each layer say whether it is config-controlled and where its rules live.

Step 3. Probe, do not grep. For every evidence command from step 1, ask each layer's own engine what it would do to that exact command (for example: rtk hook check "<cmd>"). A scan of the config file does not count; it agrees with whoever wrote the config. Record a verdict per command: passes byte-exact, rewritten, or unknown.

Step 4. Classify and fix. Navigation output (searching, listing, locating) may stay compressed. Evidence output must reach the model byte-exact: exclude it from every rewriting layer, or route it through a raw passthrough. Where one command serves both roles (grep used to certify an absence), treat that use as evidence.

Step 5. Report. Output a table: claim, evidence command, layers in its path, probe verdict, action taken. Flag any verdict-shaped output ("identical", "passed", "clean", "ok") that crosses a lossy layer as top severity, because a false success wakes no safety net. End with every command you could NOT probe, marked UNKNOWN, never assumed safe.

Shared as is, for reference. Read it and decide what it will do before you run it; using it is your responsibility, under the terms.