How I Gamify AI Collaboration Without Trusting the Score

On 27 June, I typed a gloriously unedited instruction into an AI coding session.

I wanted the model to stop occasionally, inspect how we were working and name the weak spots. My prompt was less tidy:

Proceed with both Grok and Claude local as critics, rate yourself from time to time on 1) usage of skills 2) usage of Grok and Claude local 3) implementation effectivness 4) usage of sub-agents and tools 5) effectivness of your QA and code accuracy

The reply gave me something precise to argue with.

The first scorecard exposed unfinished work

The model began on a five-point scale:

skills                 4/5
external critics       3/5
implementation         3/5
sub-agents and tools   3/5
QA and code accuracy   4/5

Each score included a gap. Critics had not run. No code had landed. Extra agents had not been justified. The baseline gate had passed, although one flaky run remained unresolved.

The integers proved nothing. The explanations gave me inspectable claims. I could ask why QA deserved a 4, reject weak evidence and turn a vague complaint into a process change.

The session gradually settled on this loop:

dimension → evidence → gap → process correction → human-checked receipt → rescore

This is one long case, not a controlled experiment. I cannot credit the score for the final result. The transcript does show what happened after I required the scorecard: weak areas were named, I demanded corrections, and the independent checks remained authoritative.

One row looked like this:

dimension: external critics
before:    5/10
evidence:  capability check passed; no fresh critic review in this loop
gap:       the changed work had not faced an independent challenge
action:    run two bounded critic passes and verify every finding
after:     9/10
receipt:   first pass found blockers; blockers fixed; second pass found
           no blockers; full test gate passed
checked by: human confirmed the critic record and full gate result

Later successful work still scored critic use 7.5 and tools 7, then named the next simplification. At least the score did not rise automatically whenever the task ended.

The game helps the human

A number adds a target and a visible next round. That makes a long session easier to re-enter. Instead of reopening the whole argument, I can ask what would move QA from 7 to 8 and what receipt would prove it.

The model does not become motivated. I get a progress log and a smaller surface for disagreement.

There is a useful human parallel. A meta-analysis of 138 randomised studies found that interventions increasing progress monitoring improved goal attainment, with larger effects when progress was recorded or reported. That evidence concerns people pursuing goals. In this workflow, the person is the one using the score to maintain attention and direct the next move.

The scorecard also makes priorities explicit. I choose the dimensions. The model shows how the current transcript supports them. If it awards 9 for QA while a browser journey never ran, the mismatch improves the rubric for the next round.

An AI self-score is generated text

The number is a diagnostic prompt output. It is not a probability, an internal reading or a ship verdict.

A June 2026 study asked six models for confidence across 12,000 medical-question responses. Accuracy did not guarantee good calibration, and calibration varied by model and specialty. The authors describe self-reported confidence as behavioural output rather than a standardised internal measure. Open-ended quality scores have fewer anchors still. The paper is narrow: medical multiple-choice questions under a fixed prompt. I use it as a warning about reading too much into a model’s number, not as a study of this workflow.

Iterative critique can improve outputs. Self-Refine reported roughly 20% average absolute improvement across seven evaluated tasks from a generate, feedback and revise loop. A TACL critical survey supplies the limit: prompted self-correction holds up mainly when the task suits it or dependable external feedback enters the loop.

Tests decide whether tests pass. A rendered journey shows whether the journey works. Sources support or break a factual claim. A critic can spot a blind spot, although its agreement remains advice until checked.

Reward-model overoptimisation research shows how an imperfect proxy can improve while target quality stalls or falls. A chat scorecard is not reinforcement learning, but, as an analogy, the warning still helps: a model can learn that 9 ends the review.

This post caught its own inflated score

I used the post as a demonstration of the ritual, then asked Grok to challenge it:

round 1, my self-score       7.5/10
round 2, my self-score       8.5/10
Grok review of round 2       6.8/10  → REVISE

Grok found the exact failure the article warned about. My prose required an “external receipt”, but the copy-paste prompt let the same model invent the evidence, approve the receipt and raise its own score.

It also rejected two lines that treated the score as the cause of later workflow changes. That was fair. The transcript establishes sequence, while critics and tests establish what improved. An integer gets no credit for the work it merely describes.

This three-round exercise does not validate the method. It demonstrates the correction loop and shows why an independent reviewer belongs inside it.

Copy the loop, including its referee

Review how you handled this work session.

I define the dimensions. Do not add, remove or rewrite them mid-loop:
- use of relevant instructions and skills
- use of tools and independent reviewers
- implementation effectiveness
- QA and factual accuracy
- communication and alignment with my goal

For each dimension:
1. Give a score from 1–10.
2. Cite concrete evidence visible in this session.
3. State the strongest counterexample or unresolved gap.
4. Name the smallest process change that could earn the next point.

If a score is below 8, apply that process change in the current loop
when feasible. Then present the external receipt for my review.

Do not raise a score until I confirm the receipt. Valid receipts include
command output, a failing-then-passing test, a critic log, a rendered
journey or a checked source. Model-written narrative is not a receipt.
For subjective dimensions, ask me whether the result improved.

Do not raise a score for effort or another attempt. Use NOT RUN when the
available evidence cannot support a score.

Keep a small ledger beside it:

dimension | before | evidence | gap | correction | after | checked receipt

Four or five stable dimensions are enough. Stop after two review rounds unless new evidence arrives. Otherwise the agent can spend the afternoon reviewing its review, an activity at which it has heroic stamina.

Where the scorecard stops

Run objective checks whenever they exist. A self-score never substitutes for the build, the browser, the source or the independent review.

Keep this separate from release approval. I use a five-axis QA gate to judge the output itself and require receipts from the real journey before calling work ready. This scorecard diagnoses the collaboration process.

The method earns its keep when a low score changes the next action and the new action leaves a receipt another person can verify. Once the score stops producing corrections, I stop scoring.

Further reading

• Does monitoring goal progress promote goal attainment?: a meta-analysis of 138 randomised human studies.
• Calibration of self-reported confidence and accuracy of LLMs: current evidence that stated confidence and accuracy are different measurements.
• Self-Refine: iterative feedback and refinement across seven tasks.
• When can LLMs actually correct their own mistakes?: a critical survey of where self-correction holds up.
• Scaling laws for reward-model overoptimisation: the proxy-measurement failure behind the Goodhart warning.

Comments & Reactions

Got a thought, a war story, or a “well, actually”? Sign in with GitHub and jump in.

Loading comments…