Your model will be retired: an ops playbook for LLM migrations

In the six weeks before this article was written, Anthropic gave Claude Opus 4.1 a retirement date (August 5, 2026), OpenAI removed the entire GPT-4.1 family from its pricing page, and xAI replaced Grok 4.1 with Grok 4.5. Three vendors, three quiet rotations, one summer. If your application hardcodes a model ID, one of these was - or will be - your outage.

A model is not a constant. It is a dependency with a deprecation calendar, and it deserves the same operational discipline as any other dependency: an inventory, a pinned version, a tested upgrade path and a rollback plan.

The two questions#

Prototype: “Does the new model work on the prompts I just tried?” Production: “Can I prove the new model is at least as good, at a cost I predicted, with a revert path if I’m wrong?”

A vibes-based swap is fine in a notebook. In production, a model change is a deploy - and it deserves a deploy’s ceremony.

What breaks in production#

• Deadline discovered as an outage. Requests to a retired model fail outright. If nobody owns the deprecation calendar, the retirement date is the day your error rate finds it for you.
• Model IDs scattered through the codebase. Ten call sites, three repos, one forgotten cron job. The migration that “took an afternoon” resurfaces for weeks.
• The API surface moved with the model. Migrations are not just an ID swap: newer models remove parameters older code depends on. Anthropic’s current models, for example, return a 400 if you set temperature, top_p or top_k - code that worked on the old model is a hard error on the new one.
• Same prompt, different bill. Newer models can tokenize differently (Anthropic documents roughly 30% more tokens for the same text on its recent tokenizer) and carry different prices. A swap that improves quality can quietly double spend - or halve it, if you never re-check.
• Nobody measured the regression. The new model is better on average and worse on your three most important cases. Without an eval gate, you learn this from customers.

The playbook#

1. Inventory before anything#

You cannot migrate what you cannot find. List every place a model ID appears: application configs, prompts-as-code repos, eval harnesses, cron jobs, notebooks that became services. Vendor consoles help - Anthropic, for example, lets you export usage broken down by API key and model, which is exactly the audit you need to catch the forgotten caller.

2. Pin explicit IDs in one place#

One config file, one source of truth, referenced everywhere:

# models.py - the only file where model ids live
MODELS = {
    "chat":       "claude-sonnet-5",       # main assistant
    "classifier": "claude-haiku-4-5",      # cheap routing
    "judge":      "claude-opus-4-8",       # eval judge
}

Aliases that silently resolve to “latest” are convenient in dev and a liability in prod: your model can change under you without a deploy. Pin, and make every change to this file go through review.

3. Gate the swap on your eval set#

This is the heart of the migration. Run your eval dataset against the new model before any traffic sees it, and diff against the old model’s baseline:

old = run_evals(model="claude-opus-4-1", cases=EVAL_SET)
new = run_evals(model="claude-opus-4-8", cases=EVAL_SET)

for metric in ("pass_rate", "faithfulness", "format_ok"):
    assert new[metric] >= old[metric] - TOLERANCE, f"regression in {metric}"

Pay special attention to the cases that made you build the eval set in the first place - the past incidents. A new model that wins on average and loses on your worst historical failure is not an upgrade.

4. Re-check the request surface#

Read the vendor’s migration notes for removed parameters, changed defaults and new stop reasons. Grep your codebase for every parameter the notes mention. This step costs ten minutes and prevents the classic post-migration 400s.

5. Canary, and watch three numbers#

Route 5-10% of traffic to the new model and watch quality (eval pass rate and hallucination rate on sampled traffic), cost (per-request spend - re-model it in the cost calculator first, then confirm against real token counts) and latency (p95, not the average). Hold the canary long enough to cover your traffic’s daily shape.

6. Keep rollback one step away#

The old config value is your rollback. Keep the previous model ID deployable until the canary has earned your trust and the old model’s retirement date has not passed - after that, your rollback target no longer exists, which is its own reason not to migrate at the deadline.

7. Own the calendar#

Vendors announce retirements in advance - Anthropic commits to at least 60 days’ notice. That is enough time to run this playbook calmly, but only if the dates live somewhere a human looks: put every model you run in a calendar with its deprecation status, and review it monthly. The incident-response template is the wrong place to first learn a model died.

Minimal vs mature#

Aspect	Minimal	Production-grade
Model IDs	Hardcoded at call sites	Pinned in one reviewed config
Swap decision	Vibes on a few prompts	Eval gate with regression tolerance
Rollout	100% at once	Canary → watch → full
Cost	Discovered on the invoice	Re-modelled before, confirmed during
Deadlines	Discovered as an outage	Calendared, reviewed monthly

Where this lives in a real system#

Model pinning and rollback are the Deployment layer of the stack; the gate that makes swaps safe is the Evaluation layer. The multi-provider gateway blueprint shows the architecture that makes switches cheapest - one routing point instead of ten call sites - and the Production Checklist has the “can we roll back a model change?” items to clear before you need them.