On this page 12 sections
A prompt is fitted to the inputs it was tested against. When those inputs drift - a new customer segment, a redesigned form, a marketing campaign that changes how people phrase things - accuracy falls with no deploy, no error and no alert. The tell is a system that was fine for months and is now “a bit worse”, where nobody can point at a change. The cause is upstream and the fix is monitoring the input distribution, not the prompt.
This is written as a pattern rather than a single incident, because it recurs with the same shape across very different systems. The details below are archetypal.
Key takeaways
- Your prompt did not break. The distribution it was fitted to moved.
- Test sets age with the traffic they were drawn from, so evaluation can pass while production degrades.
- Monitor input characteristics - length, language, category mix - not just outputs.
- The change is usually upstream and owned by someone who did not know you existed.
- Refresh the evaluation set on a cadence, or it measures a system that no longer exists.
Who this applies to
You have a production system that was performing acceptably and is now performing worse, and nobody can identify what changed.
The shape of it
The sequence is consistent.
Months one to four. A support agent is built and tuned against a year of historical tickets. It scores well on a held-out set, launches, and performs roughly as measured. Resolution rate holds. Everyone stops watching closely.
Month five. Marketing launches a campaign into a new segment - a different industry, a different country, a younger audience. Inbound volume rises, which is read as success.
Month six. Support leads mention the agent “seems worse lately”. Nobody can produce an example on request. Escalation rate is up a few points, attributed to the higher volume.
Month seven. Someone finally samples escalations and finds a pattern: a category of question the agent handles badly, phrased in a way that does not appear anywhere in the original ticket history.
The investigation finds no deploy, no prompt edit, no model version change. The system is doing exactly what it did in month one. The inputs are different.
Why evaluation did not catch it
The uncomfortable part, particularly for teams who did the evaluation work properly.
Your test set was drawn from historical tickets. It measures performance on the distribution as it was when you built it. Run it in month seven and it still passes, because it is still asking the old questions.
A held-out test set protects against changes you make. It does not protect against changes the world makes. Those are different threats and they need different instruments.
This is a close cousin of the provider-changed-the-model failure - see your provider changed the model, what broke - with the same signature: a dependency changed, nothing in your repository moved, and your monitoring was watching the wrong thing. There the dependency was the model. Here it is your users.
What actually drifts
Five things, roughly by how often they are the cause.
Vocabulary. A new segment uses different words for the same concepts. Retrieval matches less well, because your content uses the old vocabulary and so did your test set.
Question mix. The proportion of question types shifts. A category that was 3% becomes 20%, and it happens to be the one the system handles worst. Overall accuracy falls with no per-category change at all - a composition effect that is invisible unless you track the mix.
Input length and structure. A redesigned contact form removes a field, so messages arrive with less context. Or adds one, so they arrive with more boilerplate the prompt was not written to ignore.
Language and locale. Expansion into a new market brings inputs in a language your evaluation never covered, or English written by non-native speakers with different phrasing patterns.
Upstream formatting. A system feeding your pipeline changes its output shape slightly. Not enough to break parsing; enough to change what the model sees.
How to detect it early
Monitor the inputs, not only the outputs. This is the whole prevention.
Track the category mix over time. Classify incoming requests and watch the proportions. A category doubling is worth knowing about on the day rather than in a quarterly review.
Track input length distribution. A step change means an upstream form or system changed.
Track retrieval quality as a leading indicator. Falling average relevance scores mean incoming questions match your content less well than they did. This moves before accuracy does, which makes it the most useful single signal on this list.
Track the unknown rate. The proportion of requests your classifier cannot confidently categorise. Rising means new kinds of things are arriving.
Sample and read. Fifty real inputs a month, read by a person. Unfashionable, and it catches things no metric does, because a human notices “these are all asking about a product we launched in March” immediately.
Refresh the evaluation set
The structural fix, and the one that is easy to defer indefinitely.
Quarterly, add recent cases and retire ones about products or policies that no longer exist. If your category mix has moved, re-stratify to match.
Keep the old set too. Running both tells you something useful: if the system scores well on the old set and poorly on the new one, you have confirmed drift rather than degradation. Those need different responses - drift means updating content and coverage, degradation means something broke.
That two-set comparison is the cleanest diagnostic available for this failure, and it costs one extra evaluation run.
Fixing it, in order
Once confirmed, the order matters because the cheapest fix is usually the right one.
- Update the content, not the prompt. Most drift-related failures are questions your documentation does not answer. Writing the missing article fixes retrieval, the agent, and the human team at once.
- Add the new cases to the evaluation set, so the fix is measurable and the regression cannot silently return.
- Re-tune retrieval if vocabulary has shifted - synonyms, or reindexing with updated content.
- Then consider the prompt. Last, because a prompt change is global and this problem is usually local to a category.
Reaching for the prompt first is the instinct and it is usually wrong. It is the one lever that affects every case, applied to a problem affecting some cases.
What we do differently because of this
Input distribution monitoring goes in with the build - category mix, length, retrieval score, unknown rate - alongside output quality. And the maintenance retainer includes a quarterly evaluation-set refresh rather than treating the set as finished at launch.
The reasoning is that this failure is invisible to every instrument teams normally have. Uptime is fine, errors are flat, deploys are clean, and the held-out set passes. A system can degrade for a full quarter with every dashboard green, and by the time it is noticed the trail is cold - nobody remembers what changed in month five.
The habit we push hardest is the monthly read of fifty real inputs. It is the least sophisticated item on the list and it consistently finds things the metrics miss, because a person reading recognises a new pattern in a way that requires no threshold to be crossed.
Where we get this wrong ourselves: the quarterly refresh is the easiest maintenance item to defer, because nothing is broken when it is due. The refresh that gets skipped is the one that would have caught the drift, and we have had to make it a scheduled deliverable rather than a good intention.
When this is not your problem
Stable, internal, narrow inputs. A pipeline processing one document type from three suppliers does not drift much.
Very short-lived systems. A campaign tool running six weeks will not see it.
Where you control the input completely. If the input is generated by your own system with a fixed schema, drift comes only from your own changes, which your regression tests already cover.
Frequently asked questions
How do we distinguish drift from degradation?
Run the old evaluation set and a fresh one built from recent traffic. Passing the old and failing the new is drift. Failing both is degradation - something in the system changed.
How often should the test set be refreshed?
Quarterly for most business systems, plus continuously from production failures. Monthly if your traffic mix moves quickly.
What is the single best early signal?
Average retrieval relevance score. It falls before accuracy does, because retrieval is where a vocabulary mismatch first shows.
Should we retrain or re-tune when this happens?
Usually neither. Update the content first - most of these failures are questions your documentation does not answer, and the model is behaving correctly given what it was given.
Can we prevent drift?
No. You can detect it early and respond cheaply, which is the achievable goal. Systems whose inputs come from the world will always drift.
Next step
If you have no visibility on your input distribution, that is the gap this pattern exploits. The AI evaluation and QA engagement instruments input monitoring alongside output quality, and refreshes the evaluation set on a cadence.
Related: Your provider changed the model, what broke · Building an eval set from real tickets · Regression testing prompts like code · AI evaluation and QA