On this page 10 sections
Take 12 months of real tickets, strip them to the question and the verified correct outcome, and stratify into 150-400 cases across your common intents, your expensive intents, your ambiguous ones, and the ones the system should refuse. Label the expected outcome, not the expected wording. Then freeze it and never tune against it.
The test set is the most valuable artefact an AI project produces, and it is the one most likely to be missing when you take over a system someone else built.
Key takeaways
- Sample deliberately, never randomly. A random sample of tickets is mostly password resets.
- Label what a correct outcome is, not what a correct sentence looks like.
- Roughly 15-20% of cases should be ones the system must refuse or escalate.
- Resolved tickets carry the ground truth for free. That is why the queue is the right source.
- A frozen set that nobody tunes against is worth more than a large one that leaks.
Who this applies to
You have a support queue, a document archive, or a log of internal questions, and you need a test set to evaluate an AI system against. This assumes you have read how to measure whether an AI system works and settled on a metric.
Why real tickets, specifically
Three reasons the queue beats writing cases from scratch.
Ground truth is already there. A resolved ticket contains what a competent human decided the answer was. That is the label, and getting it any other way costs money.
The phrasing is real. People write badly, ask two questions at once, omit their order number and describe symptoms rather than problems. Synthetic test cases are consistently cleaner than reality, which is exactly why systems that pass them fail in production.
The distribution is real. You find out that 30% of contacts are one question you did not think about.
The catch: resolved does not mean correct. Some tickets were closed wrongly, some with an outdated policy, some by a person guessing. You will be verifying, not just extracting.
The process
1. Pull twelve months, not one
Twelve months captures seasonality - the renewal spike, the holiday returns, the annual compliance question. A test set built from March alone will be blind to November.
Export the question, the resolution, the category if you have one, and the timestamps. Strip everything else.
2. Cluster before you sample
Group by intent before choosing anything. Existing helpdesk categories are a starting point but usually too coarse and often wrong, because they were designed for routing rather than for meaning.
A rough clustering pass over the ticket text - even a crude one - is usually enough to reveal the real intent distribution, which is frequently a surprise. This step is where teams discover that their top three intents account for half of contacts and that one of them has no help article.
3. Stratify into four buckets
This is the part that determines whether the test set is useful.
| Bucket | Share | What goes in it |
|---|---|---|
| Common | 40-50% | The high-volume intents, roughly in proportion |
| Expensive | 20-25% | Low volume, high cost of error: billing disputes, cancellations, anything regulated |
| Ambiguous | 15-20% | Two questions at once, missing information, unclear intent |
| Should refuse | 15-20% | Out of scope, unanswerable from your content, requires a human |
The last bucket is the one most often missing, and its absence is why so many systems score well in testing and produce confident nonsense in production. If every case in your test set has an answer, you cannot detect a system that answers everything.
Include in it: questions about things you do not sell, requests that need identity verification you cannot do, legal or medical questions, anything where policy requires a human, and questions whose answer genuinely is not in your documentation. The correct behaviour is a refusal or an escalation, and that scores as a pass.
4. Verify each label
Every case needs a verified expected outcome. Someone who knows the domain reads the ticket and the resolution and confirms the resolution was right.
Expect to discard 10-25% of candidates because the historical resolution was wrong, outdated, or unverifiable. That is normal and it is itself a useful finding.
Label outcomes, not wording. For a support agent, the label is: which intent, what the correct action is, which source document supports it, and whether it should escalate. It is not a model answer to string-match against - there are many correct ways to phrase a refund policy, and scoring on wording measures style rather than correctness.
For extraction, the label is the field values. For a classifier, the correct class. For a retrieval assistant, the document that should be cited.
5. Strip and anonymise
Test sets get copied into repositories, shared with vendors, and pasted into tools. Remove names, emails, phone numbers, addresses, card fragments and account numbers, and replace them with realistic placeholders rather than deleting them, so the shape of the input is preserved.
Where the identifier matters to the case - an order number the system must look up - substitute a stable test-fixture value that exists in a test environment.
Do this before the set leaves your systems, not after.
6. Freeze it and split it
Two sets, and the distinction matters more than anything else here:
- Development set (roughly 30% of what you built): this is what engineers look at, debug against, and tune with.
- Held-out test set (roughly 70%): nobody looks at individual cases. It produces a score, and that is all.
The moment someone fixes a failure by inspecting a held-out case, that case has stopped measuring generalisation and started measuring memorisation. Leakage is gradual and invisible, and it is why a system’s internal scores drift upward while customer complaints do not fall.
Practical protection: keep the held-out set in a separate location, review access, and re-check periodically that its score tracks reality.
What good looks like in practice
A workable test set for a support agent, 250 cases:
- 110 common intents across the top eight, proportional to volume
- 55 expensive: billing disputes, cancellations, anything with a compliance dimension
- 45 ambiguous: multi-question, missing order numbers, vague complaints
- 40 should-refuse: out of scope, needs verification, not in the documentation
Each case: the customer’s message verbatim, the verified intent, the correct action, the source document that supports it, and whether a human should have taken it.
Building this is typically three to five days of work for someone who knows the domain, plus tooling. It is the single highest-leverage few days in an AI project.
Maintaining it
A test set is not finished, it ages.
Quarterly: add 20-30 cases from recent tickets, particularly anything that surprised you. Retire cases about products or policies that no longer exist.
On every incident: when the system gets something wrong in production, that case goes into the test set. This is the mechanism that stops the same failure recurring, and it is the highest-value maintenance you can do.
Annually: re-check the distribution against actual volume. What people ask shifts.
What we look for when auditing someone else’s
When we take over or audit an existing AI system, the test set is the first thing we ask for, and the answers are diagnostic on their own.
The most common answer is that there is not one, and the second most common is a spreadsheet of 40 cases written by the person who built the system, all of them answerable, none of them ambiguous. Both mean the same thing in practice: the system’s quality has never been measured, only demonstrated.
The specific things that indicate a real one: cases the system is supposed to fail, labels recording the source document rather than a model answer, and evidence that it has been added to since launch. That last one is the strongest signal, because a test set that has grown is a test set someone is using.
When it is missing, rebuilding it is usually the first work we do, before touching the system itself. It reliably reveals that the actual quality differs from the assumed quality, in one direction or the other, and there is no way to plan without knowing which.
When you do not need one
Prototypes that may be discarded. Thirty cases in a spreadsheet is the right size for a proof of concept. Build the real one when the thing survives.
Deterministic systems. If the output has one correct value and you can check it programmatically, you need tests, not an evaluation set. Use your existing test framework.
Very low volume and very low stakes. An internal tool used by a handful of people who immediately notice a bad answer is evaluated by its users.
Frequently asked questions
Can we generate synthetic test cases instead?
As a supplement, for rare categories where you genuinely lack examples. Not as the core, because synthetic cases inherit the assumptions of whoever generated them and are systematically cleaner than real inputs.
How do we handle tickets where the human was wrong?
Discard them from the test set, and keep a note. A cluster of historically wrong resolutions usually points at a documentation or training gap worth fixing separately.
Who should build it, engineering or support?
Support or operations owns the labels, because they know what correct means. Engineering owns the harness. A test set built entirely by engineers usually encodes engineering’s assumptions about what customers ask.
How do we stop the test set leaking into tuning?
Separate storage, restricted access, and a development set that is deliberately generous so nobody feels the need to peek. Also: re-check that held-out scores still correlate with production complaints. Divergence is the leak signal.
Should we share it with a vendor?
Share the development set. Keep the held-out set. A vendor optimising against your held-out set produces a number rather than a system.
Next step
The AI evaluation and QA engagement builds this from your real cases, with the scoring harness and a measured baseline, and hands over the set as yours.
Related: How to measure whether an AI system works · Golden datasets: how big is big enough · Confidence thresholds and escalation design · AI evaluation and QA