How to measure whether an AI system works

Measuring an AI system means four things: a metric tied to a business outcome, a fixed test set built from real cases, a baseline measured before any change, and a re-run schedule. Without a held-out test set and a baseline, “it seems better” is the only claim available, and it is not one you can act on.

Most production AI systems are evaluated by vibes and complaint volume. That works until the day it does not, and by then the regression is weeks old.

The loop runs on a schedule, not only on deploy. Running it only in CI catches your changes and misses theirs - a provider shipping a new model version triggers nothing on your side.

Key takeaways

Accuracy is almost never the right metric. Pick one tied to what the system is for.
The test set must be held out and fixed, or you are measuring your own tuning.
Measure the baseline before you change anything. It is the only comparison that means something.
Score the failure modes separately. An average hides the failures that matter.
Two hundred well-chosen cases beats two thousand scraped ones.

Who this applies to

You have an AI system running - a support agent, a document pipeline, a classifier, an internal assistant - and you need to answer “is it working” with a number rather than an impression. Also relevant if you are about to accept delivery of one and need to verify what you are being handed.

Step one: pick a metric that means something

The default answer, “accuracy”, is usually wrong because it does not distinguish between failures with wildly different costs.

Choose from what the system is actually for:

System	Useful primary metric	Why not accuracy
Support agent	Resolution rate without escalation, at a quality bar	A confident wrong answer scores as an answer
Document extraction	Field-level precision, plus % needing human correction	Whole-document accuracy hides which field fails
Classifier or router	Precision and recall per class	One rare, important class disappears into the average
Internal assistant	Answer-with-correct-citation rate	An unsourced right answer is unverifiable
Forecasting	Error against the naive baseline	Absolute error alone says nothing about value

Then add a guardrail metric that must not degrade while the primary improves. For a support agent, the primary might be resolution rate and the guardrail the rate of confidently wrong answers. Optimising a single number without a guardrail reliably produces a system that games it.

Finally, tie it to a business figure. Resolution rate matters because it maps to cost per contact and to how long a customer waits. If you cannot draw that line, you have chosen a metric that will not survive a budget conversation.

Step two: build a test set from real cases

The test set is the whole exercise. Everything else is arithmetic.

Source it from reality. Real tickets, real documents, real questions - not cases someone invented. Invented cases are systematically too clean and too well phrased.

Size: 150-400 cases for most business systems, and the arithmetic behind that range is worth seeing rather than taking on trust. This surprises people who expect thousands. Beyond a few hundred, the cost of labelling rises faster than the confidence gained, and a smaller set that is actually maintained beats a large one that rots. Rare but critical categories need enough representation to be measurable, which usually means deliberately over-sampling them.

Stratify deliberately. Do not sample randomly from your ticket queue, or you will get a test set that is 70% password resets and tells you nothing about the cases that matter. Build it as a deliberate mix: the common cases, the expensive cases, the ambiguous ones, and the ones that should be refused or escalated.

Include cases the system should fail. A test set with only answerable questions cannot detect a system that answers everything confidently. Perhaps 15-20% should be out of scope, unanswerable from your content, or requiring a human. The correct behaviour is a refusal or an escalation, and that should score as a pass.

Hold it out. Once cases are in the test set, they do not get used for tuning. The moment someone fixes a prompt by looking at a failing test case, that case stops measuring anything. Keep a separate development set for that work.

We wrote up the mechanics in building an eval set from real tickets.

Step three: decide how a case is scored

Three options, in increasing order of cost and decreasing order of speed.

Deterministic checks. Where there is a right answer - an extracted invoice total, a routing destination, a retrieved document ID - compare directly. Fast, free, unambiguous. Use this wherever the problem allows.

Human scoring. A person reads the output against a rubric. The most reliable and the most expensive. Reserve it for the baseline and for periodic audits rather than every run.

Model-based scoring. A separate model grades the output against a rubric. Cheap and repeatable, and the only practical option for scoring hundreds of open-ended answers on a schedule.

Model-based scoring has a real caveat worth stating: graders are biased, notably toward longer and more confident answers, and they are not reliable on subtle correctness. Calibrate it - have a human score 50 cases, compare to the model’s scores, and measure the agreement. If agreement is poor, the grader rubric needs work before its numbers mean anything. Re-check that calibration whenever the grading model changes.

A practical rubric is usually three or four binary questions rather than a 1-10 score. “Is it factually correct against the source? Does it cite a real source? Does it stay in scope? Would a customer need to ask again?” Binary questions produce far more consistent scores across graders, human or model.

Step four: measure the baseline before you change anything

This is the step most often skipped and it invalidates everything after it.

Run the test set against the system exactly as it is today. Write the number down. That number is what every future claim is measured against.

If there is no system yet, measure the human process the same way, on the same cases. “Our agents resolve these correctly 88% of the time and take four minutes” is the comparison that matters, and without it you cannot tell whether 71% is a success or a downgrade.

Baselines also expose an uncomfortable and useful fact fairly often: the existing process is worse than everyone assumed. That changes the business case in your favour, and you only find it by measuring.

Step five: know when a score has actually moved

A single number is not a result. Three things stop you chasing noise.

Run-to-run variance. Most of these systems are non-deterministic. Run the same test set three times without changing anything and record the spread. If it varies by four points, then a four-point improvement is not an improvement.

Segment the score. An average across categories hides the thing you need to know. A system at 71% overall might be 91% on billing and 38% on returns. The average suggests tuning; the segments tell you exactly where to work.

Read the failures, not the number. Every evaluation run should end with someone reading ten actual failures. The pattern is almost always more informative than the score, and it is where you discover that half the failures are one missing document rather than a model limitation.

What a real evaluation report contains

If you are being handed one, it should include:

The metric, defined precisely enough to reproduce
The test set: size, how it was built, category breakdown
The baseline and the current score, with the variance band
Per-segment scores
Guardrail metrics
A sample of actual failures with commentary
What changed since the last run
Cost per case, because quality that costs three times as much is a trade, not a win

A report that is one number and a chart is marketing.

How often to run it

On every change to prompts, retrieval configuration, or model version. Automatically, in CI, treated as a failing test.
Monthly as a scheduled full run, whether or not anything changed - because the model provider and your content both change without you.
Quarterly, review the test set itself. What people ask shifts. A test set from a year ago measures last year’s problem.

The monthly cadence catches the failure mode nobody expects: a system that has silently degraded while every deploy passed, because the thing that changed was outside your repository. See your provider changed the model, what broke.

Why we treat this as a product rather than a phase

Evaluation is the part of AI delivery most vendors describe and fewest hand over, and we sell it as a standalone engagement partly because of how consistently it is missing.

The pattern we see when auditing existing systems is not that teams evaluated badly. It is that they evaluated once, at launch, by hand, and never again - so the number in the launch deck is the only number that has ever existed. Six months later nobody can say whether the system got better or worse, and the honest answer is usually that nobody knows.

The reason it gets cut is that it is 15-20% of a build and it is the only line that produces no visible feature. A buyer comparing two proposals sees one that is cheaper and one that includes something they cannot picture. This is why we quote it as a named deliverable with an owner rather than as a methodology paragraph: it is easier to cut a paragraph than a line item.

The stance behind it: a system you cannot measure is a system you cannot improve or defend, and every argument about whether it is working will be settled by whoever is most senior in the room rather than by evidence.

When measuring formally is not worth it

Very low stakes and very low volume. An internal tool used by four people who can spot a bad answer immediately does not need a scoring harness. Reading the outputs is the evaluation.

Before the system does anything. Do not build an elaborate evaluation apparatus for a prototype that may be discarded next week. A spreadsheet of 30 cases is the right size then.

When there is nobody to read it. An evaluation nobody looks at is a recurring cost with no output. Fix the ownership first. For what the first year looks like when nobody measures at all, see we shipped without an eval set.

Frequently asked questions

How many test cases do we really need?

150-400 for most business systems, stratified deliberately. Enough per category to be meaningful, and small enough that you will actually maintain it.

Can we use a model to score another model’s output?

Yes, and for open-ended answers at scale it is the only practical option. Calibrate it against human scores on a subset first, and re-calibrate when the grading model changes.

What score is good enough?

The wrong question until you have the baseline. Good enough is “better than the current process at acceptable cost”, and that threshold is specific to you.

Should the vendor build our evaluation set?

They can build it; you must own it. If the test set lives only with the vendor, you cannot verify their claims or check the system after they leave.

Do we need this for a system we bought rather than built?

Yes, and it is harder, because you cannot see inside. You can still build a test set and measure outputs, which is the only independent check you have on a vendor’s dashboard.

Next step

The AI evaluation and QA engagement builds the test set from your real cases, the scoring harness and a measured baseline, wired into CI so a regression fails a build instead of reaching a customer.