Verifying AI Coding Agents: Receipts Beat Vibes

The agent sent me screenshots, and they all looked fine. That was the problem.

They were not faked. They were something more awkward: true, but incomplete. The first screen checked out. Further down, a block of right-to-left text had quietly gone unreadable in dark mode, well past the point anyone had bothered to scroll. And on iOS there was a second trap I missed at first: the simulator can keep serving an old JavaScript bundle, so the screenshot was a perfectly real photo of yesterday’s app.

That is how AI coding tends to fail in practice. Not a dramatic hallucination, but a plausible patch, a confident summary, and a piece of evidence that quietly answers a slightly different question than the one you asked. It is the same worry I keep circling back to here: how do you ship code from agents you can’t fully trust, and how do you catch your own confident-but-untested write-up before it goes out. This time the answer was a tooling one, so I went looking for a better QA-Dev tool.

Confidence needs receipts

The agent vendors already know evidence matters. Claude Code’s best-practices docs recommend giving the agent checks it can run for itself: tests, builds, screenshots, expected output. OpenAI’s Codex system-card addendum goes a step further: the model is trained to provide verifiable evidence of its work through citations of terminal logs and files, and rewarded for owning up to the things it couldn’t do. Both point the same direction.

The catch is that “I took a screenshot” only helps if the screenshot proves the right thing. Which branch was it? Which commit, platform, theme, language, simulator? Did it exercise the current app, or photograph a previous build under flattering light? On its own, a screenshot is a witness, not a receipt, and witnesses are worth cross-examining.

Give the agent handles

By QA-Dev I mean developer-facing commands that drop the app into a specific state, capture evidence, and fail when that evidence is thin. Not “please check dark mode”, but something the agent can actually run:

# Example shape — swap in your own script name and URL scheme.
PLATFORM=ios ./scripts/dev/qa.sh nav 'myapp://dev-qa?screen=feed&appearance=dark'
PLATFORM=ios ./scripts/dev/qa.sh sweep    /tmp/feed-dark 6
PLATFORM=ios ./scripts/dev/qa.sh snapshot /tmp/feed-dark-state

The exact commands will differ in your app; the shape is what carries. Set the state, capture the surface, then collect the state and logs that explain what the app thought it was doing. A boring command run the same way every time beats an eloquent paragraph about what the agent meant to check.

A useful QA-Dev command has a minimum contract:

• puts the app in the exact state under review,
• runs on the real target, not only a preview,
• captures the whole relevant screen,
• includes app state and logs,
• writes a small evidence manifest,
• fails when evidence is missing or suspicious.

A command that reports success with nothing behind it is not a QA command. It is paperwork.

Make the receipt rude

A screenshot persuades; a manifest interrupts. The trick is to make the evidence pack into something a reviewer can argue with.

The version that actually shipped in my own tooling is deliberately small. Each run produces a list of checks, every one tagged PASS, FAIL or INFO, plus a count:

type QAStatus = 'PASS' | 'FAIL' | 'INFO';

interface QAReport {
  suite: string;            // which QA command produced this
  generatedAt: string;
  context?: Record<string, unknown>;
  checks: { status: QAStatus; label: string; detail?: unknown }[];
  summary: { pass: number; fail: number; info: number };
}

PASS and FAIL are obvious. INFO turned out to be the interesting one: it is where the harness records something it observed but did not assert, the “I saw this, I’m not claiming it’s right” line. The report prints behind a grep-able tag, so the agent can quote it back and a CI step or a human can scan it without replaying anything.

That lean shape was already enough to change a review. Instead of “the agent says it checked iOS”, you get a concrete artifact tied to a run. But once I started rejecting evidence, I wanted more on the envelope, so the next iteration wraps the same checks with provenance and an explicit list of what went untested:

{
  "schema": "qa-report.v2",
  "command": "qa.sh sweep /tmp/feed-dark 6",
  "platform": "ios",
  "branch": "feature/reader-toolbar",
  "commit": "abc1234",
  "testedState": { "screen": "feed", "appearance": "dark" },
  "summary": { "pass": 7, "fail": 0, "info": 1 },
  "notProved": ["manual keyboard input path"],
  "artifacts": { "frames": ["01.png", "02.png", "03.png"] }
}

Now there are things to reject on. Wrong commit means stale evidence. Android-only means iOS is still unknown. Something in notProved that the change actually touched means it is not done. That notProved field is the one I most want a tired, future version of myself to read. Automation can inject a state and prove the renderer handled it, but that does not always prove the path a human takes. If the tool faked a search query instead of driving the real keyboard, the manifest should say so out loud. The gap is fine. Hiding it is the bug.

Capture what the user actually sees

Most visual QA is too polite. It grabs the top of one screen, in the one state the developer expected, and everyone nods at the same rectangle. The defects tend to live elsewhere: contrast drift, repeated components sliding out of the design system, a control creeping over text, spacing that only goes wrong after the second card, and almost anything below the fold.

So I stopped capturing a screen and started capturing a sweep: every key screen, in both light and dark, each as its own frame, with a manifest recording exactly what produced them.

# Capture every screen in both appearances, not one hero shot.
SCREENS=(
  "home|screen=home"
  "feed|screen=feed"
  "settings|screen=settings"
  "details|screen=details"
)
# -> /tmp/qa/<screen>-<light|dark>.png   plus report.json

This sits next to visual-regression testing, but it is not the same job. Playwright’s own docs on visual comparisons warn that screenshots differ across browsers and platforms because rendering and fonts differ. That warning is the mobile problem in miniature: bless a baseline before you have checked the platform, state and bundle were right, and all you get is very reproducible, very wrong baselines.

Native bugs need native evidence

A web preview is handy, and it is not native QA. It will not reliably catch iOS keyboard behaviour, a stale simulator bundle, safe-area issues, native text-input quirks, platform font rendering or gestures. Expo’s development-build docs spell out the split behind half of these: the native app is installed on the device while the JavaScript bundle is served separately during development. That seam is exactly where stale evidence sneaks in.

So the iOS path attaches the dev client to the current bundle before it takes a single frame. This is the kind of step a person remembers exactly once, painfully, after shipping a screenshot of the wrong build — and the kind an agent will skip with perfect manners every time. The fix is not to write a better instruction. It is to bake the step into the command so forgetting is not an option.

Tests can pass the wrong thing

The original bug was, at bottom, a contrast problem, and contrast is measurable. WCAG 2.2 puts the floor for normal text at 4.5:1, and 3:1 for large text. Product taste can ask for more; it cannot ask for less and still call the text readable.

Even good checks have limits, though, and the benchmark world has had a sobering year on exactly this point. A 2025 study with the blunt title “Are ‘Solved Issues’ in SWE-bench Really Solved Correctly?” built a differential tester and found that 7.8% of patches counted as correct while actually failing the developers’ own test suite, and that nearly a third — 29.6% — behaved differently from the reference fix despite passing. OpenAI separately announced it would stop treating SWE-bench Verified as a frontier coding metric, though for its own reasons: the benchmark had saturated and leaked into training data, and its audit found 59.4% of the hardest “unsolved” tasks had flawed tests. Two different failures, one shared lesson — a green check often means “passed the available test”, not “solved the problem”.

In UI work the intent is especially hard to pin down in a test. Does the screen still feel coherent? Is the text readable once you scroll? Did native input actually work? Did search keep the navigation stack instead of replacing it? Did the app honour the user’s appearance setting? Did the change add useful UI, or just more UI? Those still need judgement. Better evidence does not replace the judgement; it clears the fog around it.

Test the tester

Once a QA command is part of the workflow, it is production infrastructure, and if it rots, your review process rots with it while still printing green. The cheap insurance is a contract check on the harness itself: does the command still exist, still write the manifest in the expected schema, still classify a failure as a failure, still refresh the dev client before capture? It need not boot a simulator; it only needs to stop a silent harness regression from curdling into false confidence. A broken QA harness is worse than none, because no harness keeps you cautious and a broken one makes you sure.

The prompt to use before you have the tool

You do not need to build any of this to get most of the benefit today. With no custom tooling at all, you can still make the agent show its working:

Before calling this done, show your evidence.

Answer in five bullets:
1. State:     what exact app state did you test?
2. Platform:  which device, simulator, browser or native target?
3. Command:   what command produced the evidence?
4. Artifacts: which exact files prove it — screenshots, logs, state?
5. What did you NOT test?

If this is UI work, capture the full scrollable surface, not just the first viewport.
If this is mobile work, verify on the native target, not a web preview.
If any evidence is missing, say so plainly. Don't turn a missing
artifact into a confident paragraph.

It is not magic. It just moves the conversation off confidence and onto proof. The better version is to turn each of those five lines into a command the agent can run without being asked.

The takeaway

Do not build this for everything. If the agent renames a helper or fixes a small pure function, run the normal tests and move on; the ceremony should match the risk. Reach for QA-Dev handles when the change touches native UI, settings persistence, visual coherence, accessibility, locale, text direction or platform-specific behaviour — anything you would be embarrassed to defend with “the screenshot looked fine”.

The first such command feels like pure overhead. By the fifth it feels like leverage, because the agent stops needing a bespoke verification prompt at all: the repo already holds the right handle. That is the real win — the quality bar moves out of the chat and into the codebase, where it stays put.

I don’t want AI coding agents to sound more certain. I would settle for better receipts.

Further reading

• Claude Code best practices — including giving agents checks they can run.
• Codex system-card addendum — on verifiable evidence and admitting uncertainty.
• Are “Solved Issues” in SWE-bench Really Solved Correctly? — the differential-testing study behind the 7.8% and 29.6% figures.
• Why OpenAI no longer evaluates SWE-bench Verified — saturation, contamination and flawed tests.
• Playwright visual comparisons — screenshots vary by platform and environment.
• Expo development builds — the native-app / JS-bundle split that lets stale evidence through.
• WCAG 2.2 contrast minimum — the readability floor.

Comments & Reactions

Got a thought, a war story, or a “well, actually”? Sign in with GitHub and jump in.

Loading comments…