Model swap regression harness: a case set with expected decisions that blocks a retirement migration before it reaches production

Who this guide is written for.

• CTOs and VPs of Engineering who received a retirement notice for the model behind a production feature and need to know what the replacement will do differently before the date
• Platform teams that call models through an alias or a latest tag and have seen behaviour change with no deploy in their own history
• Owners of agents that book, cancel, refund or refuse on the customer's behalf, where a changed decision is a changed transaction
• Teams asked by finance to move to a cheaper model who cannot say what the cheaper model will stop doing

What a model swap changes, and why sample outputs do not show it

An agent in production is a loop of decisions: whether to call a tool, which one, with which arguments, whether to ask the user, whether to refuse, whether to stop. The text it returns is the last step. After a swap the text is what a reviewer reads, and a newer model's text usually reads better: cleaner, more confident, better formatted. The regression lives in the decisions. A thread on a developer forum in July 2026 describes a swap that silently broke an agent's cancellations; a second thread from August 2026 recounts a case where the replacement model, asked to check availability, booked the meeting on its own. In both cases the outputs looked fine and the state of the world was wrong.

Agent evaluation research arrived at the same place. Yao and co-authors (2024), who built tau-bench, score an agent by comparing the database state at the end of the conversation with an annotated goal state, and introduced pass^k, the probability that the agent succeeds on all k of k repeated runs of a task, because a single run hides inconsistency. Their best agent at the time succeeded on under 50% of tasks and had pass^8 below 25% in the retail domain. A harness that scores the final text with a rubric measures the wrong layer. It has to assert on the decision and on the end state, and it has to run each case more than once.

Outcome drift and execution drift

Two different things get called drift, and they need different controls. Outcome drift is the model producing a different decision on the same input because the model changed: a retirement migration, a swap to a cheaper tier, an alias that now points somewhere else. Execution drift is the same model identifier producing different behaviour because something behind the identifier changed. Chen, Zaharia and Zou (2023) measured the same GPT-4 endpoint in March and June 2023 and found accuracy at identifying prime numbers dropping from 84% to 51%, with instruction following degrading and more formatting mistakes in generated code. OpenAI's own documentation for the seed parameter says determinism is best effort and that the system_fingerprint field, which represents the backend configuration the model runs with, changes when the provider updates the numerical configuration of the infrastructure serving the model.

The practitioner workaround, repeated in every thread on the subject, is to log the model identifier and fingerprint the provider returns with each response and alert on change. That is a good observability control and it belongs in the tracing layer, next to cost and latency per call. A harness is a different instrument. The log says that something changed and nothing about what the agent now does differently, and it stays silent when the identifier is unchanged and the behaviour moves anyway. The harness answers that question; the logged identifier says when to run it outside the release cycle.

How the three main providers retire models

Notice periods and alias mechanics differ per provider, and the differences decide how much of a migration is under the team's control. The table summarises what each provider's documentation states at the time of writing; specific dates change and are not reproduced here.

The harness: cases, expected decisions, tolerance, threshold

A case is a recorded real interaction: the input, the tool responses replayed from a fixture so the environment does not vary, the expected decision, the expected tool calls with the arguments that matter, and the end state. Cases come from production traces, stratified so that the rare and expensive ones are present: refusals, irreversible actions, escalations to a person, edge cases from previous incidents. Each case carries a tolerance. A cancellation case tolerates no change in the decision and no change in the order identifier passed to the tool; a summarisation case tolerates different wording and asserts only that the named facts appear. Text criteria that need a judge model use one, and that judge needs its own calibration against human raters, which is the subject of a separate guide on this site and is not repeated here.

The threshold is written before the run, per case class, not after the diff is read. Classes with irreversible actions get zero tolerance: one changed decision blocks the swap. Classes with reversible outcomes get a share of cases that may change, agreed with the owner of that flow. Each case runs several times per model so the comparison is pass^k against pass^k, and a case that passes once and fails twice on the new model counts as inconsistent, which for an agent acting on a customer's behalf is a regression. Kapoor and co-authors (2024) argue that agent evaluation without cost is incomplete and that benchmarks without holdouts get overfitted; the harness reports cost and latency per case next to the decision diff, and a slice of cases is held back from anyone tuning the prompt for the new model.

• Triggers: model or version change, prompt change, tool schema or description change, and a change in the model identifier or fingerprint logged in production.
• Diff per case: expected versus observed decision, tool calls and arguments, end state, tokens and cost, latency, consistency across repeated runs.
• Gate: threshold per case class agreed in writing before the first run; crossing it fails the CI job and the swap does not ship.
• The confidence trap: the new model's text is hidden from reviewers until the decision diff has been read, so a more assured tone cannot be mistaken for a better decision.

The swap procedure and the rollback path

A retirement notice starts a procedure that already exists. The case set is frozen at the notice date and tagged with the old model's version identifier, and the old model's results on it are recorded as the baseline while the old model still answers. The replacement is run on the same frozen set; the diff report goes to the owner of each flow with the failed cases attached. Prompt changes made to fit the new model are versioned separately and run through the harness again, because a prompt fix for one case class often moves another. The swap ships behind a switch that can point back at the old model until its retirement date, so a regression found in production has a rollback that takes minutes, with no hotfix.

After the retirement date there is nowhere to roll back to, which is the argument for starting when the notice arrives rather than the week before. The procedure also names who watches the providers' deprecation pages and release notes, at what cadence, and where the next known dates are kept. A team that has done this once has the case set, the CI job and the diff format; the second migration takes a run and a diff review.

Where this goes wrong, and what we do about it.

1. 1
   The swap is validated on text quality and the decision changes

   The team compares sample outputs from the old and the new model, reads the new ones as cleaner and better structured, and ships. The cases where the agent should have asked, refused or stopped are not in the sample, or are in it but the reviewer reads the prose rather than the tool call. The cancellation that no longer happens, or the meeting that gets booked instead of checked, is found by a customer.

   What we do about it

   We record the expected decision, tool calls and end state for every case, and the diff is computed on those fields before anyone reads the text. Cases from the classes with irreversible actions are oversampled from production traces. The reviewer sees the decision diff first and the text second.

   What stays with you: case set with expected decisions and tool calls per case, and a decision diff report per swap with the failed cases attached
2. 2
   The alias moved and nobody deployed anything

   The application calls the model through an alias or a latest tag. The provider points the alias at a new release, within its stated notice period or, for backend configuration changes, with no notice at all. Behaviour changes on a day with no deploy in the team's history, and the investigation starts in the application code. Chen, Zaharia and Zou (2023) documented behaviour of the same endpoint moving substantially within three months.

   What we do about it

   We pin the model to a fixed version where the provider offers one, log the model identifier and fingerprint returned with every response into the tracing layer, and add an alert on change. A changed identifier triggers a harness run outside the release cycle, so the team sees the decision diff the same day rather than a symptom weeks later.

   What stays with you: pinned version in configuration, identifier and fingerprint fields in every trace, and an alert wired to a scheduled harness run
3. 3
   The newer model sounds more confident, so it is read as better

   One engineer runs a handful of prompts through both models and prefers the new one because it hedges less and formats better. Confidence in the text is taken as evidence of a better decision. In the case recounted by practitioners, the more assured model went a step further than asked and booked the meeting; the tone was the reason it was not questioned.

   What we do about it

   We compute the decision-level pass rate and the consistency across repeated runs per case class, with the text hidden until the diff has been read. Where the new model acts beyond what the case asked for, the case fails regardless of how the response reads, and the finding goes into the prompt work for the new model with the case attached.

   What stays with you: per-class pass rate and pass^k for both models on the same frozen case set, reviewed before any output text
4. 4
   The threshold is set after the result is known

   The harness runs, the new model passes 92% of cases, and 90% is declared acceptable in the same meeting. The 8% includes two refusal cases and a refund case, which nobody looks at individually because the aggregate cleared. The threshold has become a description of the result rather than a gate.

   What we do about it

   We write the threshold per case class with the owner of each flow before the first run, and classes with irreversible actions get zero tolerance. The CI job fails on any class over its threshold and lists the failed cases by class, so a refund case cannot hide behind a hundred passing summaries.

   What stays with you: threshold register per case class with owner and date, and a CI job that fails on it
5. 5
   The harness runs on model changes only, and a tool edit slips through

   A developer reorders the tool list, renames a parameter or rewrites a tool description during an unrelated ticket. The model's choice of tool depends on those descriptions, so the agent starts picking a different tool on the same input. No model changed, so the harness was not run, and the regression is attributed to the model at the next swap.

   What we do about it

   We hash the model version, the prompt and the tool schemas together and run the harness whenever any of the three changes, in the same CI job. The diff report names which of the three changed since the baseline, so the cause is in the report rather than in a bisect.

   What stays with you: combined hash of model, prompt and tools recorded with every harness run, and a CI trigger on any of them

What stays with you.

• Case set: real interactions from production traces with replayed tool fixtures, expected decision, expected tool calls and arguments, expected end state and tolerance per case, stratified by case class and versioned with the model, prompt and tool hash
• Threshold register: one row per case class with the tolerance, the owner and the date it was agreed, wired into the CI job as a blocking gate
• Diff report per swap: decision, tool call, end state, cost and latency differences between baseline and candidate, pass^k per class, failed cases attached, hidden output text until the diff is read
• CI job that runs the harness on model, prompt or tool change and on an identifier change logged in production, with the combined hash recorded per run
• Tracing fields for the model identifier and backend fingerprint returned per response, with an alert on change and a scheduled harness run behind it
• Retirement procedure: who watches which provider pages and at what cadence, the freeze and baseline steps at notice time, the switch that points back at the old model until its retirement date, and the rollback path

How we work.

1. 1
   1. Case classes and threshold register

   We map the agent's decisions and tools with you, name the case classes, and write the tolerance and the owner for each class before any run. Irreversible actions, refusals and escalations get their own classes with zero tolerance. This is the document the gate will enforce.

   Week 1
2. 2
   2. Case set from production traces

   We sample real interactions from your traces, stratified by class, record the tool fixtures so the environment replays identically, and label the expected decision, tool calls and end state per case with your flow owners. A holdout slice is set aside from anyone tuning prompts.

   Week 1-3
3. 3
   3. Baseline and first diff

   We run the current model several times per case to record the baseline and pass^k per class, then run the candidate model on the same frozen set. The first diff report goes to the flow owners with the failed cases, cost and latency per class.

   Week 3-4
4. 4
   4. CI wiring and production hooks

   We add the harness as a CI job triggered by the combined hash of model, prompt and tools, add the identifier and fingerprint fields to your traces with the alert and the scheduled run behind it, and put the model switch with the rollback path into configuration.

   Week 4-5
5. 5
   5. Retirement procedure and handover

   We write the procedure with your team: who watches the providers' pages, what happens at notice time, and how a swap ships and rolls back. We run one swap end to end with you on the harness and leave the case set, the register and the report format in your repository.

   Week 5-6

The rest of this cluster.

Where this turns into a service.

Where the dates and numbers come from.

• Anthropic, Model deprecations: lifecycle states, at least 60 days' notice for publicly released models, partner platforms on their own schedules, deprecated request parameters returning errors on newer models
• OpenAI, Deprecations: deprecation versus shutdown, notice of at least six months for generally available models, three months for specialised variants and as little as two weeks for previews, dated snapshots and aliases
• Google, Gemini API model deprecations: listed shutdown dates are the earliest possible, replacement models named per deprecation
• Google, Gemini models: version name patterns, stable versions usually do not change, latest alias hot-swapped on every release with two weeks' email notice for breaking changes, previews deprecated with at least two weeks' notice
• OpenAI Cookbook, Reproducible outputs with the seed parameter: best-effort determinism and system_fingerprint as the backend configuration the model runs with
• Chen, Zaharia and Zou (2023), How is ChatGPT's behavior changing over time?: same endpoint in March and June 2023, prime identification from 84% to 51%, instruction following and code formatting degrading
• Yao et al. (2024), tau-bench: A Benchmark for Tool-Agent-User Interaction: evaluation by comparing end database state with the goal state, the pass^k consistency metric, best agent under 50% success and pass^8 below 25% in retail
• Kapoor et al. (2024), AI Agents That Matter: cost as a metric alongside accuracy, holdout sets against benchmark overfitting, reproducibility of agent evaluation