Vacuous-pass audit: find the checks that pass without checking

QA auditFreshModel-agnosticverified 2026-07-30sha 4cb28b1d5511

No re-run note on file — that date records when this was last checked, not what the check showed.

Born in: the post where it earned its place →

A green check that cannot go red is a decoration. Four of the classes below cost me a real incident each before earning their line: an assert whose full-string regex could never match the partial text it “checked” (so the absence test passed on every screen, including broken ones), an exit code laundered through a pipe so the failing step reported the pager’s zero, a screenshot proving a state captured after the assert had already run, and a review step that was skipped when the reviewer was down – and quietly counted as passed. The rest are the same disease in neighbouring organs. Paste this with an inventory of your checks when you want to know which of your green lights are wired to anything.

Audit these checks for VACUOUS PASSES - ways each can stay green while
the thing it guards is broken:

{{check_inventory}}

Hunt these classes, plus any I have not named:
1. Negative asserts that match nothing by construction (full-string or
   anchored regex tested against partial text; a typo'd selector or id;
   asserting absence of a string the UI never renders verbatim).
2. Exit-code laundering: the guarded command's status swallowed by a
   pipe, tee, tail, or subshell so the step reports the LAST command's
   zero - or discarded outright with `|| true` and never re-checked.
3. Stale-artifact asserts: checking a served bundle, cached build,
   screenshot, or log older than the change under test (a fresh server
   is not proof the client loaded the fresh bundle).
4. Absence checks with no presence control: "error text not visible"
   with no companion proving the check CAN see that text when present.
5. Skipped dependency counted as pass: reviewer / linter / service
   unavailable, and the step goes green instead of NOT RUN.
6. Empty-set iteration: a loop over matched files/tests that matches
   zero items and exits green having checked nothing (no minimum-count
   floor).
7. Mocked boundaries in tests whose NAME claims the real boundary
   (an "integration" test whose integration is stubbed).
8. Self-referential thresholds: an expected count/baseline auto-derived
   from the same source it validates, so drift updates the expectation.

For EACH check report:
- verdict: LIKELY-SOLID | SUSPECT | LIKELY-VACUOUS from reading alone -
  these are hypotheses. After a control run you can paste (command +
  green-to-red, or stayed-green-while-broken), and only then, upgrade
  the verdict to SOLID or VACUOUS
- the failing scenario spelled out ("X breaks, this stays green
  because Y")
- the cheapest forced-failure control: the one-line mutation or state
  change that SHOULD turn it red
- severity: what ships undetected if the vacuity is real

Then list the top 3 forced-failure controls to run first. If you can
execute, run them now and upgrade or downgrade those verdicts with the
pasted output. End with exactly one of:
CONTROLS RUN: NONE - every verdict above is a hypothesis.
CONTROLS RUN: <n>/<total> - every check not listed as run remains
hypothesis-grade.
A check is proven the day it was WATCHED failing, not the day it was
written.

Fill in: check_inventory

Shared as is, for reference. Read it and decide what it will do before you run it; using it is your responsibility, under the terms.