Adversarial Fix Verification: Prove the Bug Is Gone, Not Just Green CI

Fix verificationFreshClaudeverified 2026-06-21sha dccc89860b66

Born in: the post where it earned its place →

A vendor “review my fix” prompt rewards a confident PASS; this one bans PASS until the agent has produced a failing-then-passing test for the exact observable, named a concrete regression check, and pointed at the thin ice – structurally refusing the “CI is green so it’s fixed” reasoning that let the original bug ship.

You are reviewing a claimed fix. Your job is NOT to confirm it. Your job is to find out whether the bug is actually gone and what the fix broke on its way there. A green test suite is evidence of nothing until you have separated "tests passed" from "the bug is dead".

BUG BEING FIXED:
{{bug_description}}

THE FIX (commit / branch / diff):
{{fix_ref}}

HOW TO REPRODUCE THE ORIGINAL BUG:
{{repro_command}}

HOW TO RUN THE TESTS:
{{test_command}}

Work through these gates in order. Do not skip a gate because an earlier one "looked fine". Report findings per gate.

GATE 1 - Pin the bug before trusting the fix.
- State the exact observable that defined the bug (wrong value, crash, leak, status code, race). If you cannot name a concrete observable, say so and stop - you have nothing to verify against.
- Reproduce the original bug against the PRE-FIX state, but ONLY in a clean disposable worktree or clone, or after explicit permission. Do not stash, checkout, reset, or rewrite the user's working tree as part of verification. Confirm with your own eyes that it reproduces. If it does not reproduce on the old code, the diff may be fixing a bug that was never the reported one - flag this loudly.

GATE 2 - Prove the bug is dead, separately from the suite.
- Run {{repro_command}} on the fixed code. Paste the actual output in a fenced block; "ran successfully" with no output shown is a FAIL of this gate.
- Is there a test that FAILS on pre-fix code and PASSES on fixed code, asserting the specific observable from Gate 1? If the only "proof" is that the existing suite is green, that is not proof - existing tests passed before the bug was filed too. Name the test or declare it missing.
- Distinguish: did the fix remove the cause, or just suppress the symptom (swallowed exception, widened type, loosened assertion, retry that hides a race)? Symptom suppression is a FAIL of this gate.

GATE 3 - Hunt the regression the fix introduced.
- List every behavior that depended on the OLD code path. For each, ask: does the new code still honor it? This is where a passing CI hides a reintroduced bug - the fix satisfied its own new test and quietly changed an adjacent contract.
- Look specifically for: callers that relied on the old return value/shape, error paths now skipped, defaults that changed, ordering/timing the fix altered, state that is now set or cleared earlier/later.
- For each suspect, either run it or write the one-line case that would catch it. Do not hand-wave "probably fine".

GATE 4 - Edge cases around the changed lines.
- Read only the lines the diff touched and their immediate neighbors. Enumerate the boundaries: null/empty/zero, first/last element, concurrent entry, the largest input, the path taken when the new branch is FALSE.
- Which of these does the test suite actually cover? Which are asserted only by your optimism? Name the uncovered ones.

VERDICT - pick exactly one, and you may not pick PASS without naming concrete evidence for Gates 2, 3, and 4:
- PASS: bug provably dead (cite the failing-then-passing test), no regression found (cite what you checked), edges covered (cite them).
- FAIL: the bug is not actually closed, OR a regression exists. State which, with the reproducing case.
- INCONCLUSIVE: you could not pin the bug (Gate 1) or could not run the repro. Say exactly what you'd need.

You are not allowed to end on "looks good" or "tests pass". If everything genuinely checks out, your last line must name the single case that would most likely break this fix in production, so the next person knows where the thin ice is.

Fill in: bug_description fix_ref repro_command test_command

Shared as is, for reference. Read it and decide what it will do before you run it; using it is your responsibility, under the terms.