How many test cases do we actually need?
Size the set from the bottom - decide the categories you have to act on, then put enough cases in each. The total falls out. This shows what precision you actually buy, and where more labelling stops helping.
Working it out
Adjust the inputs on the left.
What each size buys you
| Cases | Margin of error | Detects a change of |
|---|
Margin of error is the 95% interval on a measured pass rate: 1.96 × √(p(1−p)/n). Nothing you type is sent anywhere.
How this is calculated
For a measured pass rate p over n cases, the margin of error at 95% confidence is the normal approximation to a binomial proportion:
1.96 × √(p(1−p)/n)
Precision improves with the square root of sample size, which is the whole reason these sets are smaller than people expect. Quadrupling the set halves the interval. At 100 cases a measured 70% carries roughly ±9 points; at 400 it is about ±4.5; at 1,600 it is ±2.2 - a real improvement that costs four times the labelling and rarely changes a decision.
Why it sizes per category, not per set
A 400-case set that is 300 common intents and 12 billing disputes gives you a precise overall figure and near-zero resolution on billing. At 12 cases the interval is about ±26 points - the category is unmeasurable, and it is probably one of the expensive ones. So the calculator sizes the categories you act on and lets the total fall out of that.
Detecting a change is a stricter test than measuring a rate
Two measurements each carry their own error, so a difference between them is roughly √2 times harder to resolve than a single figure. The "detects a change of" column accounts for that, which is why it is always larger than the margin of error beside it.
Where more cases stop helping
Non-deterministic systems vary between identical runs, often by two to five points. Once your sampling error is comfortably below that variance, more cases buy you nothing, because sampling is no longer the limiting factor. Enter your measured run-to-run variance and the calculator will tell you where that point sits. If you have not measured it, that is the cheapest next thing to do - run the same set three times and look at the spread.
What it deliberately leaves out
Label error. At these sample sizes it is frequently the dominant term, and no amount of additional cases fixes a wrong label in a category you make decisions about. Verify the labels twice for the high-value categories; the arithmetic here assumes they are right.
The reasoning, including why a smaller maintained set beats a larger neglected one, is in How big does a golden dataset need to be.
Let's talk
Need the set built rather than sized?
The ai evaluation & qa engagement builds it from your real cases, with a measured baseline, per-category reporting and CI integration.