On this page 13 sections
- Key takeaways
- Who this applies to
- Why the baseline is almost never measured
- How to measure it
- Measure disagreement first
- What the comparison usually shows
- The design the comparison usually implies
- Compare on four axes, not one
- The mistake to avoid
- What we do with the result
- When measuring the baseline is not worth it
- Frequently asked questions
- Next step
Score your existing human process on the same test set, with the same rubric, before comparing anything to it. Two things usually follow: the human baseline is lower than the organisation believes, and it is not uniform - people are strong exactly where the model is weak. That second finding is more useful than the first, because it tells you what to automate rather than whether to.
“Better than a human” is not a threshold anyone has measured in most projects. It is an assumption, and it is usually wrong in both directions at once.
Key takeaways
- Measure the humans on the same cases with the same rubric, or the comparison means nothing.
- Human accuracy on repetitive work is typically well below what management estimates.
- Compare per category. The interesting result is where each is strong, not which wins overall.
- Include time and cost per case, not just correctness.
- Inter-rater disagreement sets a ceiling: if two experts disagree, no system can score above that.
Who this applies to
You are deciding whether to automate a task currently done by people - support responses, document review, triage, classification - and you need an honest comparison rather than a vendor’s.
Why the baseline is almost never measured
Three reasons, all understandable.
It feels unnecessary. Everyone knows how the process works, so the quality feels known. It is not - what is known is that it mostly functions.
It is politically awkward. Measuring a team’s accuracy before replacing part of their work is uncomfortable, and it is often read as building a case rather than establishing a fact.
It costs real time. Two people scoring 200 cases carefully is a few days of work that produces no feature.
The consequence of skipping it is that the project has no success criterion. “Better than what we do now” cannot be evaluated when nobody knows what we do now, so the decision gets made on a demo instead.
How to measure it
Use the same cases. The held-out evaluation set, unchanged. If you measure humans on different cases the comparison is worthless.
Use the same rubric. The three or four binary questions you score the system with. Is it factually correct against the source, does it cite a real source, does it stay in scope, would the customer need to ask again.
Score work already done, where you can. Historical tickets already contain human decisions - score those rather than asking people to redo work under observation, which changes their behaviour and takes longer.
Have someone other than the original handler score. Self-assessment is not measurement.
Record time as well as correctness. Minutes per case, including the wait before anyone picked it up. Response time is frequently the thing customers actually notice, and it is where automation wins by the largest margin.
Measure disagreement first
Before scoring anything, have two qualified people independently score the same 40 cases and compare.
Where they disagree, one of two things is true: the rubric is ambiguous, or the task genuinely has no single right answer. Both matter enormously.
If the rubric is ambiguous, fix it now. Otherwise every number afterwards is noise, including the system’s.
If the task is genuinely ambiguous, you have found the ceiling. If two experts agree only 80% of the time, no system can meaningfully score above 80% against either of them, and a vendor claiming 95% is measuring conformance to one annotator rather than correctness.
This step takes half a day and it prevents the most common way these projects produce meaningless numbers.
What the comparison usually shows
Two findings recur.
Human accuracy is lower than assumed. Not because people are careless, but because the measured conditions are the real ones: end of shift, twelfth similar ticket, ambiguous request, policy that changed last month. Management estimates come from thinking about the task carefully, which is not the state in which it is performed.
This is uncomfortable and it usually improves the business case rather than damaging it.
Performance is not uniform, and the profiles differ. This is the finding worth having.
| Typically stronger | Typically weaker | |
|---|---|---|
| People | Ambiguity, judgement, unusual cases, knowing when something is off, emotional context | Consistency at volume, recall of rarely-used policy, fatigue, response latency |
| Models | Consistency, speed, availability, perfect recall of provided content | Genuine ambiguity, cases unlike anything they have seen, knowing what they do not know |
They are close to complementary. Which is why the useful output is rarely “automate this” or “do not automate this” - it is which categories go which way, and where the handover sits.
That is the same argument the escalation gate implements. See confidence thresholds and escalation design.
The design the comparison usually implies
Because the profiles are complementary, the honest output of a baseline is rarely a single verdict. It is a split.
The pattern that falls out of most comparisons:
Categories where the system beats the baseline convincingly automate, with monitoring. These are usually the high-volume, well-documented, low-ambiguity intents.
Categories where they are close automate with a tighter confidence gate, so the uncertain cases still reach a person. You capture the volume and keep the judgement.
Categories where humans win clearly stay with people, and the system’s job there is preparation rather than resolution - retrieving the relevant policy, drafting a starting point, summarising the history. That is often worth more than it sounds: the same measurement usually shows that a large share of human handling time is gathering context rather than deciding.
That third row is the one teams miss. A baseline comparison framed as “can the system replace this” produces a yes or no. Framed as “where does each perform better”, it produces a design where the system does the retrieval and the person does the judgement, which beats both pure options on cost and quality at once.
Compare on four axes, not one
| Axis | Why it matters |
|---|---|
| Accuracy, per category | The headline, and the least uniform |
| Time per case | Includes queue wait, which is often the customer’s actual experience |
| Cost per case | Loaded human cost against model plus review cost |
| Consistency | Variance across cases and across people, which is itself a quality |
Consistency deserves attention because it is invisible in an average. A team averaging 85% where individuals range 70-95% behaves differently from a system at 82% with narrow variance - the first has a worst case that reaches customers and the second does not.
The mistake to avoid
Do not compare a tuned system to an untrained human, or a fresh human to a system on its worst day.
Specifically: measure the human process as it actually runs, not as it would run with better training and unlimited time. And measure the system after tuning but on held-out cases it has not been fitted to. Anything else compares an idealised version of one to a realistic version of the other, in whichever direction suits the person running the comparison.
What we do with the result
We run the baseline in the prove phase, before any production commitment, and we present the two numbers side by side per category rather than as a single comparison.
The reason is that the per-category split changes the shape of what gets built. A system that beats the human baseline on four of nine categories is not a failure - it is a specification. Those four automate, the rest route to a person, and the resulting design is better than either the full-automation version or the do-nothing version.
The uncomfortable part we insist on presenting: where the baseline comes back low, that is a finding about the current process, not a licence to skip quality work on the new system. Being better than a 62% baseline is not the same as being good, and a system at 71% still produces wrong answers for three customers in ten. We have had clients read a low baseline as permission to lower the bar, and it is worth naming that in the room.
We have also had baselines come back high enough to stop a project. A human process running at 94% on well-documented cases is a hard target, and the honest recommendation there is usually to automate the queue and routing rather than the judgement.
When measuring the baseline is not worth it
When the current process does not exist. A new capability has nothing to compare against; measure against the business requirement instead.
When the human process is provably broken. Six-week backlogs make accuracy academic - the constraint is throughput.
When volume is tiny. Under a few hundred cases a month, a formal comparison costs more than it informs.
Frequently asked questions
How many cases do we need to score by hand?
The same 150-400 as the evaluation set. If that is too much, score a stratified 100 and accept wider confidence intervals - stated, not hidden.
Will the team resent being measured?
Handle it directly. Frame it as measuring the process rather than individuals, report in aggregate, and share the finding that the profiles are complementary - which is usually true and reassuring, because it points at automating the tedious categories rather than the skilled ones.
What if two experts disagree on the correct answer?
That is the finding. Measure the disagreement rate first; it caps what any system can score, and a vendor claiming to exceed it is measuring something else.
Should we include response time?
Yes, and include queue wait, not just handling time. It is often where automation wins most clearly and it is frequently omitted because it makes the human comparison look worse.
Does the baseline need re-measuring?
Annually, or after a significant process or policy change. Baselines drift, usually upward as the easy work is automated away and people are left with the hard tail.
Next step
The baseline is a few days of work and it is what turns “is this good enough” into a question with an answer. The AI evaluation and QA engagement measures it as part of establishing the baseline, per category.
Related: How to measure whether an AI system works · Building an eval set from real tickets · Confidence thresholds and escalation design · AI evaluation and QA