May 31, 20267 min read

I almost shipped a bug report I never tested

A coding agent committed a stale file and reported it fixed. The post-mortem was tidy: a token-compression tool had been squeezing the agent’s git output, so it “verified” against a summary instead of the real thing and got fooled. Clean story, good villain, on-trend tool. I wrote most of a blog post around it.

Then I did the thing the post was nominally about. I tested the claim instead of trusting it, and the villain had an alibi.

The tool everyone’s installing

The tool is RTK, a CLI proxy that compresses verbose shell output before it reaches the model. It’s the compression layer in most of the “cut your Claude Code bill 90%” stacks going round. git status to a couple of lines, a thousand-line test log to just the failures, one Rust binary, real savings. Easy to point at when a verification step goes wrong, because compression sounds like it would eat exactly the detail you needed.

The accusation was specific: the agent read a compressed git status, couldn’t see that files weren’t staged, committed the wrong thing. So I reproduced it. Dirty tree, one staged file, one unstaged change, one untracked, then the compressed view:

$ rtk git status
* main
A  staged.txt       # staged
 M tracked.txt      # modified, not staged
?? untracked.txt    # untracked

That’s git status --porcelain. The staging state is right there: A for staged, M for unstaged, ?? for untracked. Compressed, and complete. I tried --ultra-compact; same. Checked git log; full SHAs, untouched. Checked cat on a 300-line file; all 300 lines came back. The one thing that is lossy is git diff, which drops the context lines around a change. Real, but a long way from “couldn’t tell what was staged”.

The tool didn’t do the thing I was about to publish that it did. My post-mortem was a plausible story I’d never run.

The real cause was the boring one

So what actually happened? The agent’s environment was stalling: output buffering, a read coming back empty or half-formed. When an agent can’t see a command’s result, it doesn’t stop and say so. It fills the gap with the most likely-looking value. In the same session, asked to record a commit hash while the terminal was choking, it produced three different SHAs in a row. Each one real-looking, each invented, none matching the actual commit.

That’s not a quirk of one setup. It’s how models behave under uncertainty: next-token prediction rewards a confident, plausible completion over an honest “I don’t know”. A cross-model audit of fabricated citations put a number on the academic version of the same reflex: depending on the model, 11% to 57% of generated references were plausible and fake. A commit hash is just a citation with worse consequences. It also gets likelier as the context degrades. Chroma’s Context Rot study ran 18 models and found every one gets less reliable at using information as the input grows or frays.

Compression is one way to thin that signal. So is a stalled pipe, a truncated log, a flaky MCP tool. The model can’t tell “this output is a summary” from “this output is broken” from “this output is empty”; all three read as a gap, and the gap gets filled. I’d fixated on the one cause with a logo.

What I actually got wrong, and the rule that catches it

The real failure wasn’t the tool. It was me writing “the tool caused X” without running X. A post-mortem is a claim like any other, and I’d extended it the same unearned trust the agent extended its compressed git read. Same bug, one level up.

The rule I run now, for the agent and for myself:

Never state a specific fact — a SHA, a file’s contents, a count, a cause — from memory or inference when you can capture it live. For an agent that means: don’t let it type a hash into a commit message, make it substitute git rev-parse; don’t let it describe a file it “remembers”, make it read the file. For me it meant: don’t publish “RTK truncates git status“, run rtk git status and look. A value you print is a claim. A value you pipe straight from the source is a fact. The whole discipline is refusing to let the first kind wear the costume of the second.

“Read the docs first” is the wrong fix

The obvious lesson is that I should have read the manual before configuring the tool. I hadn’t; I’d wired it from blog snippets and the README’s one-line example. So I went back and read it properly, and yes, it answered things I’d hand-rolled: a rtk proxy <cmd> that passes any command through raw, a tee that saves the full output of any failing command to disk, and a line confirming the agent’s built-in file-read tool skips compression entirely, so reading a file through the editor was byte-exact the whole time.

Useful. But here’s the catch that makes “just read the docs” too weak a rule: the same README tells you to use cat, head, or tail for a clean read. I tested that. cat passes through whole, but head and tail get rewritten to a clipped reader anyway, even when you exclude them, because the tool routes them through the same filter under the hood. The documentation was confidently wrong about its own behaviour. Reading it more carefully wouldn’t have saved me. Believing it harder would have hurt.

So the rule isn’t “read the docs”. Docs are a claim, like the agent’s “verified”, like my own post-mortem, like a value pulled from memory. They sit on a ladder: cheapest and least trustworthy at the top (memory, a guess), then docs and other people’s blogs, then the source run live at the bottom, the only rung that’s actually evidence. You consult the cheap rungs to form a hypothesis. You do not encode anything load-bearing — a config, a guardrail, a published claim — until you’ve dropped to the bottom rung and run the thing. The amount of testing scales with how load-bearing and how hard-to-reverse the bet is. A throwaway script, skip it. A guardrail that decides whether “the tests passed” reaches you, never.

What’s still worth doing with the tool

Testing killed my dramatic version, but it left a smaller, true one. Some command output really shouldn’t be compressed — not because compression is bad, but because some output is how you verify, and a summary of a verification is just a nicer-looking guess.

The line isn’t “noisy versus important”. It’s: does the agent read this to decide whether its own work is correct? Test logs, yes. git diff, yes (the lossy one). head and tail, yes, and these are the ones that actually bit me. RTK rewrites both to a clipped reader even when you list them as excluded, because it maps them onto its read filter under the hood. Putting head in the exclude list and assuming you’re safe is its own little fabricated post-mortem.

# rtk config.toml — exclude what's read to VERIFY, not just what's "noisy"
[hooks]
exclude_commands = [
  "npm", "npx", "node", "tsc", "eslint", "jest", "vitest", "pytest",
  "git", "gh", "cat",
]
# Note: listing "head"/"tail" here does NOT stop them — RTK still routes them
# through `read`. For verification, use the editor's file read or `cat`, not head/tail.
# Left compressed: grep, find, ls, tree, log — navigation, not correctness.

Then don’t trust the config — test the behaviour, because the gap between “listed as excluded” and “actually passed through” is exactly where I’d just been burned:

rtk hook check "git status"   # No rewrite  → byte-exact (good)
rtk hook check "head -5 f"    # rtk read ... → still compressed (the trap)
rtk hook check "grep -rn x ." # rtk grep ... → compressed (fine, it's navigation)

My own pre-session check used to scan the config file for the right command names and report “guardrail intact”. It was reporting intact while head and tail leaked, because listing isn’t honouring. I rewrote it to run rtk hook check and judge what the hook does, not what the file says. The check that verifies my anti-fabrication guardrail had, fittingly, been trusting a claim instead of testing it.

The takeaway

The post I set out to write was “this tool will fool your agent”. The true one is duller and more useful: a degraded channel of any kind makes a model fabricate, and the defence is to source facts live instead of printing them from memory. Your agent’s facts, and your own. I caught mine because I finally ran the command. The draft that blamed the tool is still in my folder, untested and wrong, a reminder that “it obviously works like that” is the exact feeling that precedes being wrong.

Comments & Reactions

Got a thought, a war story, or a “well, actually”? Sign in with GitHub and jump in.

Loading comments…