Model Upgrades Break Prompts: Build the Regression Suite First

A provider announces that the model version you pinned will be retired, or a newer version is cheaper and faster, and someone on the team says the obvious thing: change the model string and see what happens. Nothing in your test suite fails, because nothing in your test suite calls the model. The deploy goes out on a Tuesday.

Two weeks later support forwards a screenshot. An invoice summary now says the total in a different format, or the agent that used to look up an order before answering now answers from memory, or the JSON your parser expects arrives wrapped in a friendly sentence. Each of these is small. Together they are the reason people say prompts are fragile, when the real problem is that the prompt was never under test.

A model version is a dependency, and a very large one with behavior you do not control. You would not upgrade your database driver without running the tests. The same rule applies here, but you have to build the tests first, and the usual instinct of writing a dozen clever examples by hand does not get you there.

What actually changes when the version changes

It helps to know what you are looking for. In our experience the differences between two versions of the same model family fall into a few buckets, and almost none of them are about whether the new model is smarter.

• Format drift. Markdown headers where there was plain text, a preamble before the JSON, a different date format, bullets instead of a paragraph. Anything downstream that parses or displays the output is exposed.
• Verbosity and tone. Answers get longer or shorter. A support reply that was three sentences is now seven, and your UI truncates it mid-thought.
• Tool selection. The new model may call a tool more or less eagerly, pass arguments in a slightly different shape, or skip a lookup it used to make. If you have an agent, this is the bucket that costs money.
• Refusals and caution. Borderline inputs that used to be answered are declined, or the reverse. This varies by domain, and it is hardest to predict for anything medical, legal or financial.
• Token usage. A different tokenizer or a more talkative model changes the cost per request, sometimes by enough to matter in a monthly budget.

Notice that a single benchmark score tells you about none of these. Public leaderboards measure general ability. Your regression suite measures whether your product still behaves like your product.

Build the suite from production, not from imagination

The cases that break are the ones you did not think of, which is why a hand-written set of examples gives false comfort. Start from real traffic instead.

Pull a sample of real requests from your logs, stratified by feature and by input type, so that the rare but important paths are represented. A few dozen cases per feature is a useful start, and a couple of hundred across the whole product is plenty to detect a broad regression. Then add every input that has ever caused a complaint or a manual correction. Those are your most valuable cases because they sit exactly where the model is shaky.

Two cautions. First, scrub customer data before it enters a test fixture, or generate a faithful synthetic equivalent; a regression suite that carries real customer documents around in a repository is its own incident waiting to happen. Second, store the expected outcome with each case, not just the input. Where you have human-reviewed answers from your own review queue, use them. If you are working without labels, our post on evaluating an AI feature with no labelled dataset covers how to bootstrap them cheaply.

Three kinds of check, in order of trust

1. Hard assertions

These are plain code and they never flake. Does the output parse against the schema? Did the agent call the expected tool? Is the extracted total equal to the sum of the line items? Are all the cited document ids ones the user was allowed to see? Write as many of these as you can, because they give you a pass or fail with no judgment involved.

2. Field-level accuracy against reviewed answers

For extraction and classification, compare fields to the reviewed answer and report a rate per field. A drop from one version to the next on the date field only is much more useful than a single overall number, and it tells you which prompt line to fix.

3. Judged rubrics, used sparingly

For open-ended text you may need a second model to grade against a rubric. This is the weakest of the three because the judge has its own variance. Use it for relative comparison, run old and new answers side by side with the order randomized, and never let it be the only gate on something that matters.

The comparison procedure

The mechanism is simple: run the same cases through the old and new version under the same prompt, and diff the results. Do not change the prompt and the model in the same step, or you will not know which one caused the movement.

1. Freeze the suite and tag it with a version, so results are comparable over time.
2. Run it against the current production model and record the baseline, including cost and latency, not only correctness.
3. Run it against the candidate with the prompt unchanged. Record the diff.
4. Triage each regression: a real quality loss, a format change you can absorb, or a case whose expected answer was wrong.
5. Only now adjust the prompt for the candidate, and re-run both versions to confirm the old one has not regressed.

A small harness is enough. The shape below is all you need to start, and the important part is that the gate is a rule written down before you see the results.

const baseline = await runSuite(cases, { model: PINNED })
const candidate = await runSuite(cases, { model: NEXT })

const report = compare(baseline, candidate)
// report.hardFailures: cases that passed before and fail now
// report.fieldDeltas:   per-field accuracy change
// report.costDelta:     tokens per case, candidate vs baseline

const ship =
  report.hardFailures.length === 0 &&
  Object.values(report.fieldDeltas).every((d) => d > -0.02) &&
  report.costDelta < 1.25

if (!ship) process.exit(1) // block the model change, not the deploy

The thresholds above are placeholders. Pick yours from how much each field matters to the product, and write them into the repository so a change to a threshold is a visible diff.

Roll the new version out like any risky change

A passing suite means the known cases are fine. It does not mean production is fine. Pin the model version explicitly in configuration, never to a floating alias, so an upgrade is always a deliberate commit. Then ship it behind a flag, route a small share of traffic or one friendly tenant to the candidate first, and compare live metrics such as parse failures, escalation rate and cost per request.

Sampling a slice of live traffic through both versions in the background, without showing the candidate output to anyone, is a good way to extend the suite with fresh cases. Cost is the thing to check here, since the candidate may be cheaper per token and still more expensive per task. Our note on caching and prompt reuse explains why the second run of a stable prompt prefix is often the cheapest tokens you will ever buy, and a version change can quietly break that reuse.

When this is more process than you need

If your product has one prompt, a human reads every output before it goes anywhere, and a bad answer costs a minute of editing, a full suite is overkill. Keep ten saved examples, run them by eye on upgrade day, and move on. The same goes for a prototype that nobody depends on yet.

Small suites also have limits. With thirty cases, a one-case change is a three percent swing, and you cannot tell noise from signal. Be honest about that in the report, and prefer hard assertions to percentages until the set grows. The suite becomes worth its upkeep when the AI feature is on a critical path, when outputs are parsed by code, or when an agent can take actions with side effects.

If you are shipping an AI feature and want the evaluation harness built alongside it rather than after the first incident, that is part of how we approach AI and agentic systems, and the way we scope that work is on our approach page.