Regression testing prompts like code

Treat a prompt change like a code change: it lives in version control, it runs against a held-out test set in CI, and it does not merge if the score drops beyond the known run-to-run variance. The specific failure this prevents is the common one - someone fixes a complaint by adding an instruction, and nobody notices the nine cases it broke.

Prompts are the only part of a production AI system routinely edited without review, testing or a rollback path. That is a process gap, not a technical limitation.

Key takeaways

Prompts are logic. Editing them in a vendor console is editing production without a diff.
Establish run-to-run variance before you can judge any change.
Gate on per-category scores, not the average - most regressions are local.
Every fixed production failure becomes a test case, or you will fix it again.
Pin the model version in CI, or you cannot tell your regression from theirs.

Who this applies to

You have prompts in production and more than one person can change them. This assumes you have an evaluation set - if not, build one first, because everything here depends on it.

Why prompt edits are uniquely dangerous

An instruction added to fix one case applies to every case.

Someone complains that the agent is too verbose on billing questions. Add “be concise”. Now it truncates the multi-step return instructions that needed to be long. The billing complaint is resolved, the returns experience is worse, and nothing connects the two events - they are separated by weeks and reported by different people.

This is not a rare pathology. It is the normal life cycle of an unmanaged prompt, and it is why prompts that have been edited for a year are usually a pile of contradictory instructions nobody dares remove.

Two properties make it worse than an equivalent code change. Effects are non-local - there is no call graph showing what an instruction touches. And effects are probabilistic - the same prompt produces different outputs, so a regression appears as a shifted rate rather than a broken test.

The setup

1. Prompts live in the repository

Not in a vendor console, not in a database row edited through an admin UI. Files, in git, deployed like everything else.

That gives you a diff, an author, a message explaining why, review, and a revert. If your platform requires prompts in its console, keep the repository as the source of truth and deploy from it, so at minimum the history exists.

Store them as plain files with the variables interpolated at runtime - a templating format your language already has is fine. Avoid embedding prompts in long string literals inside application code, purely because diffs become unreadable and unreadable diffs do not get reviewed.

2. Establish variance before anything else

Run your evaluation set three to five times against the unchanged system and record the spread.

That number is your noise floor. If repeated runs vary between 68% and 72%, a change scoring 71% has done nothing measurable, and treating it as an improvement is how teams convince themselves a rewrite worked.

Re-establish it whenever the model version changes. Variance is a property of the model, not of your prompt.

3. Wire it into CI

On any change to a prompt file, retrieval configuration or model version:

Run the held-out set against the changed system
Compare to the current baseline, overall and per category
Fail the build if any category drops by more than the variance band
Post the per-category comparison on the pull request

The per-category gate is the important part. An average is a poor detector here: a change that improves the largest category by four points and destroys a small one by thirty can leave the average flat. Most real regressions are local, and local is exactly what an average hides.

4. Every production failure becomes a case

When something goes wrong in production, the fix is two commits: the prompt change, and the test case that would have caught it.

Without the second, the same failure returns after the next refactor. With it, your test set grows in exactly the direction your system actually fails, which is a better sampling strategy than any amount of upfront design.

The gate only works if variance was measured first, because a score that moves by two points on identical runs cannot block on a one-point drop. The return path at the bottom is what makes the set grow toward the failures that actually happen.

What to gate on

Check	Gate	Why
Overall score	Drop beyond variance	Blunt, catches catastrophes
Per-category score	Drop beyond variance in any category	Catches the local regression
Refusal rate	Moved materially either way	A prompt that stops refusing is a real risk
Output format validity	Any failure	A parse error is a hard break
Cost per case	Increase beyond a threshold	Longer prompts and outputs cost money
Latency	Increase beyond a threshold	Verbosity has a user cost too

The last two are worth including even though they are not quality. Prompt changes routinely add tokens, and a change that improves accuracy by a point while adding 40% to cost is a decision somebody should make deliberately rather than discover in a bill.

Pin the model in CI

If your test pipeline uses a floating model alias, you cannot distinguish your regression from the provider’s. A failing build tells you something changed and not what.

Pin the version in CI. Upgrade it deliberately as its own change, with its own test run and its own commit - which also gives you a clean before-and-after when a provider ships something new. See your provider changed the model, what broke.

When the score legitimately drops

Not every regression should block. Sometimes a change trades one thing for another deliberately, and the gate needs an override that leaves a record.

Three cases where accepting a drop is correct.

A deliberate quality-for-cost trade. Routing simple cases to a cheaper model may cost a point of accuracy and half the spend. That is a business decision, and the person approving it should see both numbers.

A tightened refusal policy. Making the agent refuse more often on regulated topics will lower the resolution rate. The score falls and the system improved.

A test set that was wrong. Occasionally a failing case reveals that the expected answer was mislabelled. Fix the label, not the prompt - and note it, because a pattern of “fixing” the test set to pass is how a gate becomes decorative.

The mechanism that keeps this honest is requiring an explicit override with a written reason, recorded on the merge. A gate anyone can silently bypass is not a gate; a gate nobody can bypass gets removed the first time it blocks something urgent. The override with a reason survives both.

The prompt that grew

A pattern worth naming, because the fix is counter-intuitive.

Prompts accumulate. Each instruction was added to fix a real case. After a year there are forty of them, several contradict each other, and nobody will remove any because nobody knows which are load-bearing.

The test set is what makes removal safe. Delete a group of instructions, run the set, look at the score. Often nothing moves - the instruction was fixing a model quirk that no longer exists, or duplicating something the model does anyway. Occasionally the score drops sharply and you have learned which instruction was doing the work.

Do this in small batches with a run between each, or you learn nothing about which one mattered. Half a day of this on a year-old prompt typically removes a third of it, and shorter prompts are cheaper, faster and less likely to conflict with the next addition.

What we require

Prompts in the client’s repository, an evaluation run in CI on every change, a per-category gate, and a variance figure established before any of it. This is in the build, not a later hardening phase.

The reasoning is about who edits prompts. They are the most accessible part of the system and therefore the most edited, frequently by whoever is nearest the complaint. That is reasonable - the person hearing the customer should be able to act - and it is only safe when a bad change fails a build instead of reaching production.

The rule we argue for most is the per-category gate, because it is the one that gets softened under delivery pressure. An overall gate is easier to pass and it approves precisely the changes you most want to catch. We would rather ship a week later than merge a change that improved the average by trading away a category nobody was watching.

The honest limitation: this catches regressions against cases you have. A change that breaks something absent from your test set passes cleanly. That is an argument for growing the set from production failures, not an argument against the gate.

When this is over-engineering

A prototype with one editor. You are the review process.

A prompt that has never changed and works. Add the test set before you first need to change it, not before that.

Very low stakes with immediate feedback. An internal tool whose users would notice a bad answer within minutes has a human regression test already.

Frequently asked questions

How many test cases does CI need?

The same 150-400 stratified cases you evaluate with. If the full run is too slow or costly for every commit, run a stratified subset per commit and the full set nightly.

What does a CI evaluation run cost?

Usually a few dollars per run at a few hundred cases, depending on model and prompt size. Cheap against the cost of a regression reaching customers. Track it, because it grows with the test set.

How do we handle non-determinism in CI?

Set temperature low for evaluation runs, and gate against the variance band rather than an exact figure. Do not chase determinism you will not have in production.

Should prompts be reviewed like code?

Yes, by someone who understands the domain rather than only the code. A prompt change is a policy change, and the reviewer should be able to say whether the new instruction is correct, not just whether it parses.

What about prompts embedded in a no-code platform?

Keep the repository as the source of truth and export or deploy from it. If the platform genuinely cannot support that, you have accepted production edits without history, and should at least record changes elsewhere.

Next step

If prompts are being edited in a console without tests, that is the highest-leverage process fix available. The AI evaluation and QA engagement builds the harness and wires it into CI so a regression fails a build.