Field-test a model swap before you trust the changelog

Model migrationModel-agnosticupdated 2026-08-28sha efbbe3e4c660

A new model lands, the changelog says it is better at everything, and the migration is one string. Then something answers as the old model because a fallback you forgot about is still wired, or a refusal arrives as a successful 200 and your error handling never sees it. The interesting failures are not in the benchmark table. This forces the boring receipts: prove which model answered, re-run the prompts that already hurt, and only then talk about quality.

A model is being swapped from {{old_model}} to {{new_model}}. Treat this as a behaviour change, not a version bump. Your job is to find what MOVED, not to confirm the new one is better.

PROMPTS THAT ALREADY GIVE US TROUBLE: {{worst_prompts}}
HOW TO RUN THINGS: {{verify_cmd}}

Work in order. Report each step's result before starting the next. Paste real output, never a description of output.

STEP 1 - Prove which model actually answered.
- Send one trivial request to each model id and capture the FULL response metadata, not just the text.
- Show the field that names the responding model. If the response does not carry one, say so - that is a finding, and it means every later step is attributing behaviour to an id you cannot verify.
- Check for silent fallback: is there any config, gateway, router, or default that could serve a different model than the id you sent? Name each one you found and how you ruled it out.
- If you cannot establish which model answered, STOP. Everything below is worthless without this.

STEP 2 - Re-run the prompts that already hurt.
Not the showcase ones. The ones with long context, ambiguous instructions, strict output formats, refusal-adjacent content, and heavy tool use.
- Run each on both models. Same input, same parameters, same temperature.
- Run each at least twice per model. A single pair cannot separate a real difference from sampling noise. Say how many runs each result rests on.
- Report per prompt: same / different / unstable-on-both.

STEP 3 - Check the failure shapes, which is where migrations actually break.
For each, state what the new model does and whether your code handles it:
- REFUSAL: does a refusal arrive as an error, or as a successful response whose body is a refusal? Code that only checks the status will treat the second as success.
- TRUNCATION: what happens at the output limit - a clean stop reason, or a sentence that just ends? Does your code see the difference?
- TOOL CALLS: same schema, same argument encoding, same behaviour on a malformed tool result?
- LONG INPUT: does behaviour change past some context length, and where?
- STRICT OUTPUT: does it still honour the format instruction, or has it started adding a preamble?
Any row where your code cannot distinguish the new behaviour from success is a finding. Give the file:line that would need to change.

STEP 4 - Say what you did not test.
List the prompt shapes, tools, languages, and input sizes this exercise never touched. This list is the real risk, and it is the step people skip.

RULES.
- A difference is a difference only if it survives repeats. Report anything seen once as UNSTABLE, not as a regression.
- Never quote a benchmark, a changelog, or a release note as evidence about your system. The only admissible evidence is output you produced from your own prompts.
- Separate "the new model is different here" from "the new model is worse here". You are measuring the first. Only say the second when you can point at a concrete requirement it now fails.
- Cost and latency are behaviour too. Report the change, or state that you did not measure it.

END WITH: go / no-go on the swap, the single behaviour most likely to break in production, and the code change that must land first.

Fill in: old_model new_model worst_prompts verify_cmd

Shared as is, for reference. Read it and decide what it will do before you run it; using it is your responsibility, under the terms.