On this page 11 sections
When a provider updates a model, working systems change behaviour without any deploy on your side. The failures are usually not dramatic: output formatting drifts, refusal behaviour shifts, tone changes, and structured output that always parsed starts failing occasionally. Without a scheduled evaluation set, you find out from customers.
This is the most common cause of “it used to work” in production AI, and it is the one that ordinary software monitoring is worst at catching.
Key takeaways
- Your dependency changed and your version control shows nothing. That is the whole problem.
- Structured output and refusal behaviour are the two most fragile things across versions.
- Pinning a model version buys you time, not safety - pinned versions get deprecated.
- The detection mechanism is a scheduled evaluation run, not an alert on errors.
- Budget for at least one forced migration a year.
Who this applies to
Anyone running a production system against a hosted model API. Applies whether you built it or bought it, though if you bought it you have fewer levers and should read the vendor questions at the end.
What actually changes
Providers ship improvements. The improvements are usually real. They are also changes to a component you depend on, and they arrive without a pull request.
Formatting drift. The most common and most disruptive. A model that reliably returned bare JSON starts wrapping it in a code fence, or adds a sentence of preamble. If your parser was tolerant, nothing happens. If it was strict, a percentage of requests start failing, and because it is a percentage rather than all of them, it looks like an intermittent bug rather than a change.
Refusal behaviour shifts. Safety tuning moves between versions. A model that answered a category of question starts declining it, or the reverse. For a support agent handling anything medical, financial or account-security related, this shows up as a sudden change in escalation rate with no code change.
Verbosity and tone. Newer versions often produce longer, more hedged answers. Harmless in a chat UI, and a real problem when the output feeds a fixed-width field, a downstream parser, or a channel with a character limit. It also raises your token bill.
Instruction-following changes. A prompt tuned against one version encodes workarounds for that version’s quirks. When the quirks are fixed, the workaround becomes the bug. Elaborate prompts are more fragile here than simple ones.
Tokenisation and context handling. Occasionally the same input yields different truncation behaviour, which changes what the model actually saw.
The pattern, as it usually presents
It rarely arrives as an outage. The sequence is almost always:
- Nothing is deployed. Everything is green. Error rates are normal.
- A support lead mentions the agent “feels off” this week. No one can point at a case.
- Escalation rate is up four points. Attributed to seasonality.
- Two weeks later, someone finds a batch of malformed outputs in a downstream system.
- Investigation finds the provider shipped a version update on the Tuesday of week one.
The two-week gap is the cost, and it is almost entirely a detection problem. Every conventional signal - uptime, error rate, latency - looked fine, because the system was working correctly. It was just doing something slightly different.
Why standard monitoring misses it
Application monitoring is built to detect failure. This is not failure. Requests return 200. Latency is normal. Nothing throws.
What changed is the content of successful responses, and the only way to monitor content is to compare it against known-correct outcomes. That is an evaluation set, run on a schedule, treated as monitoring rather than as a testing activity.
This is the argument for evaluation that lands with engineering teams when the quality argument does not: it is the only observability you have on the behaviour of a dependency you do not control.
What to actually do
Pin the version where you can
Most providers offer dated or pinned model identifiers alongside a floating alias. Use the pinned one in production. A floating alias means you have agreed to accept behaviour changes silently, which is a strange thing to agree to for a production dependency.
Pinning is not permanent safety. Pinned versions get deprecated on a published schedule, typically with months of notice, and then you must move. What pinning buys is the ability to choose when you migrate rather than discovering that you have.
Run evaluation on a schedule, not just on deploy
Running the test set in CI catches your changes. It does not catch theirs, because nothing triggers it. A monthly scheduled run - and weekly for anything high-stakes - is what closes that gap. Alert on the score moving beyond the known run-to-run variance band.
If you do not have a test set yet, building one from real tickets takes a few days and this is one of several reasons to do it.
Make parsing tolerant, then validate strictly
Two layers. Parse permissively - strip code fences, tolerate preamble, find the JSON object within a larger string. Then validate the parsed result strictly against a schema and fail loudly if it does not conform.
Permissive parsing absorbs cosmetic drift. Strict validation ensures that a genuine semantic change is caught rather than silently accepted.
Log the model version with every request
Cheap, and it turns a forensic investigation into a database query. When quality moves, you want to answer “what version served these responses” in a minute rather than a day. Include it in your evaluation reports too, so every score is attributable to a specific version.
Keep a migration runbook
When you must move versions, the sequence should be written down rather than improvised:
- Run the full evaluation set against the new version, offline
- Compare per-segment, not just overall - migrations often improve the average and degrade one category
- Fix the prompt regressions against the development set
- Shadow-run in production if you can: serve the old version, log the new one
- Roll out behind a flag, monitor, keep the ability to revert
Budget a week for a significant migration on a mid-sized system, and expect at least one forced migration a year.
What this looks like from the buying side
If you bought a platform product rather than building, you have less control and it matters more that you ask.
Four questions worth putting to a vendor:
- Do you pin model versions, and how do you decide when to move?
- How do you detect quality regressions after a provider update? “We monitor closely” is not an answer. “We run a regression suite weekly” is.
- Will you notify us before a model migration affecting our workflows?
- Can we run our own evaluation against your system? If the answer is no, you have no independent check on quality, and you should factor that in.
A vendor with good answers here has been through it. A vendor who has not considered it will discover the problem on your traffic.
What we changed after seeing this pattern
We now treat scheduled evaluation as part of the maintenance retainer rather than as an optional add-on, and version logging is in the default build rather than something added when it is needed.
The reasoning is that the failure mode is structurally invisible. Every other class of production problem announces itself - something errors, something slows, something stops. This one produces a system that is working, responding, and quietly worse, and the only mechanism that surfaces it is a comparison against known-correct answers on a timer.
It also changed how we write prompts. Prompts that lean on a specific model’s quirks perform better today and break harder later. Simpler prompts with the complexity moved into retrieval and validation survive migrations with less rework, and we take that trade even when it costs a point or two of benchmark performance.
When to not worry about this
Low-stakes internal tools where a human reads every output and would notice. The evaluation apparatus costs more than the failure.
Fully deterministic pipelines that use a model only for a step you already validate programmatically. Your existing validation catches it.
Very short-lived systems. A campaign tool running for six weeks will probably not see a migration. Note the “probably”.
Frequently asked questions
How often do providers actually change models?
Frequently enough to plan for. Major providers ship new versions several times a year and deprecate older ones on published schedules. Treat at least one forced migration a year as a planning assumption.
Does pinning cost more?
Not usually in price. It costs a little in capability, because you are not getting improvements automatically, and it costs a scheduled migration when the pin expires. Both are worth it for production.
Can we self-host to avoid this entirely?
You remove the involuntary changes and take on the infrastructure, the engineering time, and the responsibility for security updates. It is a real option at high steady volume and an expensive one below that. You also still have to manage your own upgrades.
How big a score drop should trigger action?
Anything beyond your measured run-to-run variance. If repeated runs vary by three points, treat a five-point drop as real. Establish that band before you need it.
Should we test on the new version before it is default?
Yes, when providers make preview versions available. Running your evaluation set against a preview is cheap and turns a forced migration into a planned one.
Next step
If your system has no scheduled evaluation, that is the gap this article is about. The AI evaluation and QA engagement builds the set, the harness and the schedule, so a provider change fails a test instead of reaching a customer.
Related: How to measure whether an AI system works · Building an eval set from real tickets · Hallucination rate: what number is acceptable · Maintenance and support