Golden datasets: how big is big enough

150-400 cases is right for most business systems, and the reason is arithmetic rather than convention. At 100 cases a measured 70% carries a confidence interval of roughly plus or minus 9 points; at 400 it is about plus or minus 4.5. Going to 1,600 buys you plus or minus 2.2 - a real improvement that costs four times the labelling and rarely changes a decision. Size the per-category counts, not the total.

The question is almost always asked about the total. The total is not what limits you.

Key takeaways

• Precision improves with the square root of sample size. Quadrupling halves the interval.
• Your smallest category sets your real resolution, not your headline count.
• A 250-case set that grows from production failures beats a 2,000-case set that rots.
• Run-to-run variance often exceeds sampling error, so more cases stop helping.
• Label quality dominates below a few hundred cases.

Who this applies to

You are building an evaluation set and deciding how many cases to label, or reviewing one you inherited and wondering whether it is large enough to trust.

The arithmetic

For a measured pass rate, the margin of error at 95% confidence is roughly:

1.96 × √(p(1−p)/n)

At p = 0.7, that gives:

The practical reading: below about 150 cases your measurement is too coarse to detect the changes you care about. Above about 400, sampling error stops being your limiting factor - something else is.

Your smallest category is the real constraint

The number that matters is not the total. It is the count in the smallest category you need to make decisions about.

A 400-case set that is 300 common intents and 12 billing disputes gives you a precise overall figure and near-zero resolution on billing. At 12 cases the interval is roughly plus or minus 26 points - the category is unmeasurable, and it is probably one of the expensive ones.

So size from the bottom: decide the categories you need to act on, put 30-50 cases in each, and let the total fall out. For eight categories that lands at 240-400, which is where the general guidance comes from.

Where a category genuinely matters and examples are rare, over-sample it deliberately. Your test set does not have to mirror production frequency - it has to give you resolution where decisions get made. Just record that you did it, so nobody reads the overall figure as a production estimate.

What one case actually contains

Sizing arguments are easier to have than labelling, so it is worth being concrete about the unit being counted. A case is not a question and an answer.

For a support agent, one case is:

• The input, verbatim. The customer’s actual message, misspellings and all.
• The category, from your own stratification scheme.
• The expected outcome - which intent, what action is correct, whether it should escalate.
• The supporting source - which document or record contains the answer, so you can measure retrieval separately from generation.
• Any context the system would have had - the customer’s tier, their recent orders, whatever the live system sees. A case evaluated without the context production would supply is measuring a different system.
• A note on why this case is in the set, especially for the awkward ones. Six months later nobody remembers why a strange case was included, and undocumented cases get deleted during a cleanup.

That last field costs a sentence and prevents the slow erosion of exactly the difficult cases the set exists for.

The labelling cost follows from this list rather than from the count. Cases needing a decision about the correct action take minutes each; cases where the historical resolution is clearly right take seconds. A set weighted toward ambiguous and expensive categories - which yours should be - costs more per case than the average suggests, and that is the real reason to size deliberately rather than aim high.

Where sampling error stops mattering

Three things typically dominate before you reach 400 cases.

Run-to-run variance. Non-deterministic systems vary between identical runs, often by two to five points. When that exceeds your sampling error, more cases do not make the measurement more useful. Measure your variance and stop growing the set once sampling error is comfortably below it.

Label quality. A 1,000-case set where 8% of labels are wrong is worse than a 250-case set that is carefully verified - the errors are systematic rather than random, so they do not average out. Below a few hundred cases, label quality dominates every other consideration.

Distribution mismatch. A large set drawn from last year measures last year’s traffic precisely. Precision about the wrong thing is not an improvement, and this is the failure mode of the big set that nobody refreshes.

Grow it in the right direction

The right size at month twelve is not the right size at launch, and the growth should not be random.

Every production failure becomes a case. This is the highest-value growth mechanism there is: it samples exactly where your system actually fails, which no amount of upfront design achieves.

Add cases for new categories as your product changes.

Retire cases about products, policies or processes that no longer exist. Test sets are usually allowed to grow and never to shrink, which is how they end up measuring a system that no longer exists.

A set that goes from 250 to 400 over a year through production failures is far more valuable than one that started at 400 and never moved, even though the counts end up similar.

Auditing a set you inherited

Taking over a system usually means taking over its test set, and the count is the least informative thing about it. Four checks, in order.

Per-category counts. Not the total. If the smallest category you make decisions about has under 20 cases, the set cannot tell you anything about it, and that is often where the expensive failures are.

Are there cases that should fail? A set where every case has a correct answer cannot detect a system that answers everything. If the should-refuse bucket is missing, the set has never measured the failure mode that matters most.

Has it grown? Check the history. A set that has not changed since launch is measuring the system as it was designed rather than as it fails, and the absence of added cases usually means production failures were never fed back.

Was it leaked? Harder to establish. The signal is a held-out score that has drifted upward while complaint volume has not fallen. If you find test cases quoted in prompt files or in tuning notes, treat the score as unreliable and rebuild.

An inherited set that fails two or more of these is usually faster to rebuild than to repair, because the labels themselves are then suspect.

A worked sizing example

For a support agent with eight intent categories, where three are high-value:

Labelling that is roughly three to five days for someone who knows the domain. That is the actual cost, and it is the reason the number is not 2,000.

What we build and what we push back on

We build 200-400 case sets, sized from per-category counts, and we treat growth from production failures as part of the maintenance retainer rather than a one-off.

The pushback we give most often is against clients who want a much larger set, usually because a larger number feels more rigorous. The honest response is the curve above: past a few hundred cases you are buying precision you will not act on, while the labelling cost is linear and the maintenance burden grows. A 2,000-case set that nobody refreshes measures a system you no longer run.

The opposite pushback matters too. Where a client wants 60 cases because labelling is tedious, that set cannot detect a five-point regression, which means it cannot do the job it exists for. Under about 150 the exercise produces a number without producing information.

The thing we insist on that costs time: verify the labels twice for the high-value categories. Label error is the dominant term at these sample sizes, and a wrong label in a category you make decisions about is worse than a missing case.

When a smaller set is fine

Deterministic checks. Where correctness is programmatically verifiable - extracted totals, routing destinations - you are writing tests, not sampling, and coverage matters more than count.

Prototypes. Thirty cases in a spreadsheet is right for deciding whether an approach is viable.

Very narrow scope. A system handling three intents needs three well-covered categories, not a broad set.

Frequently asked questions

Is 100 cases ever enough?

For a rough go/no-go, yes, with the interval stated - roughly plus or minus 9 points at 70%. Not enough to detect the five-point regressions that matter in production.

Should the set mirror production frequency?

No. Mirror it for the common categories and over-sample the rare, expensive ones, then report per category. An overall figure from a deliberately stratified set is not a production estimate and should not be presented as one.

How much does labelling cost?

Three to five days of domain-expert time for 250-400 cases, including verification. The verification is the part that gets cut and the part that matters most.

Can we generate cases synthetically?

For rare categories where real examples are genuinely scarce, as a supplement. Not as the core - synthetic cases inherit the assumptions of whoever generated them and are systematically cleaner than real inputs.

How often should we refresh it?

Continuously from production failures, plus a quarterly review of whether the category distribution still matches reality.

Next step

If you have a set and do not know its per-category counts, that is worth checking before trusting any figure it produces. The AI evaluation and QA engagement builds the set sized from the categories you actually make decisions about.

Related: Building an eval set from real tickets · How to measure whether an AI system works · Testing an AI system against the human baseline · AI evaluation and QA