Measuring cost per resolved task

Divide total cost by tasks actually resolved, not by tasks attempted. Include the model spend on the failures and the human minutes spent on escalations, because both are costs of the system. A rising total with a falling unit cost is a business growing. A rising unit cost is a system degrading, and it is the only one of the two that needs action.

Most AI reporting shows monthly spend and deflection rate on separate charts. Neither answers whether the thing is getting cheaper or more expensive per unit of work done.

Key takeaways

The denominator is resolved tasks, not requests. Failures cost money and resolve nothing.
Human review time is part of the unit cost. Excluding it flatters the number substantially.
Deflection rate rewards refusing to escalate, which is the wrong incentive.
Compare against the fully-loaded human cost of the same task, not against zero.
Track the distribution, not just the mean - a small tail of expensive tasks moves it.

Who this applies to

You have an AI system in production handling a repeatable unit of work - a support conversation, a document, a classification - and you need a number that tells you whether it is improving.

The formula

Cost per resolved task = (model spend + infrastructure + human review time) ÷ tasks resolved without human completion

Each term has a trap.

Model spend includes the failures. A task that consumed three retries and then escalated cost you four model calls and resolved nothing. Its spend belongs in the numerator and it contributes nothing to the denominator - which is the whole point of the metric.

Infrastructure is the retrieval index, hosting, observability and scheduled evaluation runs, allocated across tasks. Small per unit, and it stops the metric from drifting when someone adds a component.

Human review time is the one that gets omitted, and it is frequently the largest term. If 35% of conversations reach a person and each takes six minutes at a loaded rate, that dominates the model spend by an order of magnitude. A cost-per-task figure without it is not measuring the system, it is measuring the API bill.

Tasks resolved means completed to the quality bar without a human finishing the job. Not “the bot sent a message and the conversation ended”.

The highlighted term is the one that is frequently the largest and most often missing. A figure without review time measures the API bill, not the system.

Why not deflection rate

Deflection rate is the standard metric and it has a structural flaw: it improves when the system escalates less, regardless of whether that was correct.

Tune the confidence gate looser and deflection rises immediately. So does the wrong-answer rate, but that appears weeks later as repeat contacts and complaints, attributed to something else. You have optimised a dashboard.

Cost per resolved task is harder to game in that direction, because a wrong answer that produces a repeat contact adds a task to the numerator’s costs without adding one to the denominator. It still needs a quality guardrail beside it - see hallucination rate: what number is acceptable - but it does not reward the specific bad behaviour that deflection rate rewards.

Counting resolution honestly

The definition of “resolved” is where these numbers get quietly inflated, exactly as it is in vendor contracts.

Reasonable definition: the task was completed to the quality bar, no human finished it, and the same requester did not return within 24 hours with substantially the same request.

That last clause matters more than it looks. Without it, a system that half-answers and produces a second contact books two resolutions for one job. With it, the metric reflects work actually completed.

Where you can ask, a satisfaction signal is stronger still. Where you cannot, the repeat-contact window is a good proxy and is available in every helpdesk.

Track the distribution

The mean hides the thing you can act on.

Cost per task is usually heavily skewed: most tasks are cheap, and a small tail is dramatically more expensive - long conversations, documents that triggered many retries, agent runs that looped near the ceiling.

Look at the median and the 95th percentile alongside the mean. If the mean is well above the median, a small number of expensive tasks is driving your bill, and finding those is far more productive than optimising the common path. Frequently they share a cause: one intent that retrieval handles badly, or one document type that fails validation repeatedly.

Attributing the shared costs

Model spend is easy to attribute if you tag requests. The shared lines are where the number gets fudged, and there are three of them.

The retrieval index. One index frequently serves several workflows. Allocate by query volume rather than splitting evenly - a workflow issuing 80% of the queries should carry 80% of the cost, or you will conclude the wrong system is expensive.

Evaluation runs. A real recurring cost and a genuinely shared one. Allocate to the system being evaluated; where a run covers several, split by case count.

Engineering time. The largest and the most often excluded. Maintenance, incident response and the monthly evaluation review are costs of running the system. Excluding them produces a unit cost that looks excellent for a system consuming two days a month of someone’s attention.

The rule that keeps this honest: if the cost would disappear when you switched the system off, it belongs in the number. That test resolves most allocation arguments quickly, and it is the one that pulls engineering time in.

You do not need precision here. An allocation that is roughly right and consistently applied lets you compare month to month, which is the whole purpose. An allocation that changes whenever the number is inconvenient does not.

The comparison that matters

The number is meaningless alone. Compare against the fully-loaded human cost of the same task.

Fully loaded means salary, employment costs, tooling, management overhead and unproductive time - typically well above the raw hourly rate. Divide by tasks actually completed per hour, including the queue time.

Then the comparison is honest: this task costs $0.34 automated and $4.10 by a person, at a quality level of X against Y. Both halves needed - a cheaper task done worse is not a saving, it is a different product.

What moves it

Lever	Effect on unit cost	Risk
Reduce retrieved context	Large	Quality, must be measured
Route easy tasks to a cheaper model	Large	Misrouting, evaluate per tier
Cache repeated requests	Medium	Staleness
Reduce escalation rate correctly	Large	Only if quality holds
Reduce retries	Medium	None, this is waste
Improve source content	Large, indirect	None, helps everything

That last row is the one teams skip because it is not technical. A large share of escalations are caused by content that does not answer the question, and writing the missing article reduces cost, improves quality and helps humans at the same time. See cutting LLM costs without degrading quality.

What to put in a monthly report

Four lines, and no more:

Cost per resolved task, with median and 95th percentile
Resolution rate at the quality bar, per category
The guardrail metric - unsupported-claim rate, or whatever you chose
The human-baseline comparison, refreshed quarterly

Lines one and two together are the whole story: falling cost with holding quality is progress; falling cost with falling quality is a trade someone should approve; rising cost with rising quality may be fine.

Reporting spend alone invites the wrong conversation, which is “why is this getting expensive” rather than “is it earning its keep”.

What we report and what we refuse to

We report cost per resolved task with human review time included, and resolution counted with the 24-hour repeat window. Both choices make our numbers look worse than the alternatives, and both are the reason the number is usable.

Excluding review time is the standard flattering move in this category. It typically halves the figure or better, and it produces a metric that improves when a system escalates more - which is precisely backwards. We would rather present a larger honest number than a smaller one that rewards the wrong behaviour.

The metric we decline to lead with is deflection rate, and this occasionally causes friction because it is what clients have been shown by previous vendors and what their board is used to seeing. The argument we make: deflection rises the moment you loosen the gate, so a rising deflection chart is equally consistent with a system getting better and a system getting careless. We will report it as a secondary line, next to the guardrail, never on its own.

Where our own claim is weakest: the human-baseline comparison depends on a loaded cost figure the client provides, and those vary widely in how they are calculated. We use the client’s number and state which one, rather than picking a favourable industry average.

When this is not the right metric

Where tasks are not comparable. If one unit is a two-line question and another is a multi-day investigation, a per-task average is meaningless. Segment first.

Very low volume. Under a few hundred tasks a month the number is noise.

Where the value is not cost. If the system exists for response speed or 24-hour coverage, measure that. Cost per task is the right metric only when substitution for labour is the point.

Before there is a quality bar. Cost per resolved task without a definition of resolved is cost per task attempted with extra steps.

Frequently asked questions

Should failed tasks count in the numerator?

Yes. They consumed money and resolved nothing. Excluding them measures the cost of your successes, which is a different and less useful number.

How do we value human review time?

Fully loaded cost divided by productive hours, using your own figures. Include the queue wait if you are also comparing response time.

What is a good cost per resolved task?

There is no external benchmark worth quoting - it depends entirely on task complexity and your labour costs. The only meaningful comparisons are against your own human baseline and against your own figure last quarter.

How often should we measure it?

Monthly for reporting, and on every significant change as part of the evaluation run so a cost regression is caught with the quality regression.

Does this work for document processing?

Yes, with resolved meaning posted without human correction. The review change rate is the natural quality guardrail beside it.

Next step

If your reporting is total spend and deflection rate, switching to a unit metric usually changes what the team works on within a month. The AI cost audit attributes spend per workflow and establishes the unit figure.