September 1, 20267 min read

My token compressor said two different files were identical

On 6 August 2026 an agent of mine compared two golden files as part of a PR’s evidence chain. The tool said “Files are identical”. A byte-compare in node said they weren’t, and the byte-compare was right.

The tool wasn’t diff. It was a token compressor wearing diff‘s clothes.

The setup

I run rtk (Rust Token Killer, Apache-2.0) as a Claude Code PreToolUse hook. It rewrites Bash commands into compressed equivalents before the output reaches the model: grep becomes rtk grep, find becomes rtk find, and the model reads a fraction of the bytes. The write-ups that convinced me to install it advertise 60–90% savings. I recommend the tool; across 54,565 wrapped commands on my machine it has saved 53.2 million input tokens.

My lifetime number, though, is 13.2% saved, not 60–90. Some of that gap is workload mix, and the numbers aren’t measuring the same thing anyway. But part of it is deliberate, and the deliberate part is this post.

First, a confession: my guardrail began as a myth

Back in May 2026, shortly after installing the hook, an agent committed stale golden files and reported “done”. I blamed the compressor. Obviously compressed git status had hidden staged-vs-unstaged state, so I wrote a strongly-worded exclusion list keeping git byte-exact, and felt responsible about it.

Weeks later I actually tested what the hook does to git output. In my tests, porcelain status codes came through intact and log output kept full SHAs. My incident story was false. The stale commit traced to the model fabricating values while a pipe stalled, which is a real failure needing a different rule entirely: never emit a SHA or a count from memory, capture it live and inject it. I’d built a guardrail against a behaviour the tool didn’t have, because the story was plausible and testing felt unnecessary. It wasn’t the first time; see I almost shipped a bug report I never tested.

I kept the git exclusion anyway. Verification output seemed like a bad place to accept transforms I hadn’t audited, even after the audit cleared the specific one I’d feared.

Then the guard itself sprang a leak

The config excluded head and tail by name, so evidence reads like head -5 fixture.json would pass through raw. They didn’t. The router matched bare head but sent head -5 file through a separate read filter, which compresses. And my checker blessed it: I had written a scan that grepped the config for the exclusion entries, which were of course present, since I’d just written them. It agreed with me by construction. It could not fail at the layer that mattered.

What works is asking the hook engine what it would actually do, command by command:

for c in 'git diff' 'diff a b' 'head -5 f' 'npm test' 'grep -rn foo src'; do
  printf '%-22s ' "$c"; rtk hook check "$c" 2>&1
done

rtk hook check prints the rewrite it would perform, or No rewrite for: …. Any command you treat as evidence that prints a rewrite is a live hole. The first run of this loop caught the head -5 leak my static scan had blessed; the fix was rtk’s transparent_prefixes = ["head ", "tail "]. The loop now runs in a pre-session flight script, because configs drift and hooks update.

Then the incident I’d wrongly imagined happened for real

One layer over from where I’d been looking. The agent ran diff on two golden files inside a live PR’s evidence chain, the hook rewrote it, and the compressed output read:

Files are identical

They were different. What failed here wasn’t recall or nuance. A comparison primitive returned a false verdict, and a false “identical” is success-shaped: every safety net keyed on failure, from retries to a human glancing at red text, stays asleep. The agent had exactly what it needed to write “verified, goldens unchanged” into the PR, and the claims built on that line inherit the error. diff and cmp joined the byte-exact list that day, with the reason recorded where the next agent will read it.

Fairness requires a retest, so I re-ran the experiment on rtk 0.45.0 while writing this. On two files differing by one byte, rtk diff no longer claims identity. It prints both files in full, which fails loudly and is, pleasingly, more tokens than the raw diff. The bug I hit is gone; the category of bug is the point.

Compress navigation, never verification

The sort that fell out of those three scars, and the rule I now apply to any layer sitting between a tool and a model:

Navigation output helps the agent find where to work: grep, find, ls, tree. Compress it. A dropped match can still misroute the agent, but the miss tends to surface at the next step, not inside a shipped claim.
Verification output is the evidence behind an assertion: test runners, git status and git diff, CI results, diff/cmp, the cat of a file about to be certified. This must reach the model byte-exact, because a lost or rewritten line here becomes a wrong claim with your agent’s name on it.

Command names are only a proxy for that split. The same grep that navigates can also certify an absence (“no more callers of the old API”), and at that moment it is verification. rtk’s escape hatch for exactly this is rtk proxy <cmd>, a raw passthrough that keeps tracking; I checked it against raw diff output on 0.45.0 before writing this sentence, byte-identical. Claude Code’s native Read/Grep/Glob tools bypass the Bash hook entirely, which takes this layer out of their path, though every reading tool has its own limits and the habit worth keeping is knowing what sits in each channel, not canonising one of them.

In config, the rule is an exclusion list. Mine, lightly trimmed, ledger comments intact:

[hooks]
# VERIFICATION GUARDRAIL — a summary of a verification is a
# nicer-looking guess. If the agent reads output to judge its own
# work, that output must be byte-exact.
exclude_commands = [
    "npm", "npx", "pnpm", "yarn", "node",
    "tsc", "eslint", "prettier",
    "jest", "vitest", "pytest", "mypy", "ruff",
    "cargo", "go", "playwright",
    "git", "gh",
    "cat", "head", "tail",
    # diff joined 2026-08-06 after a LIVE wrong verdict:
    # compressed diff printed "Files are identical" on two files
    # a node byte-compare proved different. A comparison that can
    # report equality falsely must stay byte-exact forever.
    "diff", "cmp",
]
transparent_prefixes = [
    "head ",
    "tail ",
]

I keep the incident ledger in the file itself because the next session’s agent reads the config, and it should inherit the scar tissue, not just the setting.

The bill is real. The excluded commands produce the bulkiest output, so this list forfeits a lot of the advertised saving; my 13.2% is what’s left on my workload, and I consider it good money for being able to believe my own agent’s “done”. If a human reads every output before anything is claimed, you can afford to compress far more, though the agent still routes itself mid-task on what it reads, so the boundary matters less there rather than not at all.

The wider boundary

rtk is where I paid the tuition, and it happens to be the layer easiest to fix, because it has a config. The same question applies to layers that don’t. Anthropic’s context editing clears old tool results; its compaction API replaces earlier history with a summary; OpenCode prunes old tool output past a token budget; research like Squeez prunes tool output task-conditionally, and Slipstream studies validating compaction for long-horizon agents. Clearing stale exploration is mostly clearing navigation. But if an agent later cites an early test run as evidence, that evidence is now a summary, and what I want to know of any summariser is whether “all gates passed earlier” can come out of it when they didn’t. That is the same question my diff answered badly.

I went looking for this framing in the write-ups I read while checking my facts, across English, Japanese, Chinese, and Korean. They go deep on mechanisms and savings, and they weigh cost against quality. I didn’t find one that sorts the bytes by what the agent is about to stake a claim on. That is the split I actually needed.

So, next time you sit down with an agent: list the commands and tool results your agent uses as evidence for “done”, “passed”, “identical”, “clean”. Find out what sits between each of them and the model, with a probe that can fail, not a config grep. Compress the rest. I’ve packaged the audit as a copy-paste prompt in the verification-channel-audit, in the library.

Related: Your agent says the tests passed. It didn’t run them. is the sibling failure, evidence never gathered; this post is evidence gathered and then rewritten on the way in. Pin the bytes is the same instinct applied to dependencies.

Comments & Reactions

Got a thought, a war story, or a “well, actually”? Sign in with GitHub and jump in.

Loading comments…