On this page 10 sections
Model confidence is a weak escalation signal on its own, because language models are confidently wrong in exactly the cases you most want to catch. A working escalation gate combines retrieval quality, topic rules, conversation state and customer signals - and it is tuned against the cost of each error type, not against a single accuracy number.
The gate is the product. Everything else in a support agent is plumbing.
Key takeaways
- A model’s stated confidence correlates poorly with correctness. Do not gate on it alone.
- Retrieval quality is a better signal than model self-assessment: no good source, no answer.
- Some topics should escalate regardless of confidence. Encode those as rules, not scores.
- Tune the threshold against the relative cost of a wrong answer versus an unnecessary handover.
- A repeat contact within 24 hours is the strongest escalation signal most teams do not use.
Who this applies to
Anyone designing or tuning an AI system that must decide between answering and handing over - support agents, triage systems, document pipelines with a review queue, internal assistants.
Why model confidence is not enough
The intuitive design is: ask the model how sure it is, escalate below a threshold. It does not work well, for three reasons.
Self-reported confidence is poorly calibrated. Ask a model how confident it is and you get a number that is plausible, stable, and only loosely related to whether the answer is right. Calibration is checkable and rarely checked - see better decisions, not just faster ones for the one query that settles it. The failure cases that matter most - a fluent, well-structured, entirely fabricated policy - often come with high stated confidence, because fluency and correctness are different things.
Token probabilities measure the wrong thing. Where you have access to them, low probability indicates the model was unsure which words to use, not that it was unsure of the facts. A confidently wrong answer can have high token probability throughout.
A single number cannot encode policy. “Never discuss another customer’s account, no matter how confident you are” is not a threshold. It is a rule.
Confidence still has a place, as one input among several. It should not be the gate.
The signals that actually work
| Signal | What it catches | Strength |
|---|---|---|
| Retrieval quality | Questions your content cannot answer | Strong |
| Topic and category rules | Cases policy requires a human for | Absolute |
| Repeat contact | The agent already failed this customer | Strong |
| Conversation turn count | Going in circles | Moderate |
| Sentiment and escalation language | Frustration, threats to leave | Moderate |
| Action risk tier | Irreversible or financial operations | Absolute |
| Model confidence | Genuine ambiguity, sometimes | Weak alone |
Retrieval quality is the best single signal
For any system answering from your own content, the most reliable question is not “how sure is the model” but “did we find anything good to answer from?”
Concretely: the relevance score of the top retrieved passage, the gap between the top result and the next, and whether the retrieved content actually addresses the question rather than merely sharing vocabulary with it.
If retrieval comes back weak, the model is about to answer from general knowledge rather than from your documentation. That is precisely the condition that produces confident, plausible, wrong answers about your refund policy. Refusing to answer when there is no good source removes an entire class of failure, and it is a cheap check.
This is the single highest-value change we make to systems that were gating on model confidence alone.
Topic rules are absolute, not probabilistic
Some categories go to a human every time regardless of any score:
- Anything legal, medical, or safety-related
- Account security, identity changes, password resets involving verification
- Complaints that reference regulators, lawyers or public complaint channels
- Financial disputes above a threshold you set
- Anything involving a minor, or a vulnerable-customer indicator
- Cancellation and retention, if that is a commercial decision
These are policy, and they belong in code as deterministic rules rather than in a prompt as a request. A prompt is an instruction the model usually follows. A rule is one it cannot bypass.
Repeat contact is underused
If the same customer asked something similar in the last 24 hours and it was handled by the agent, escalate immediately.
This is easy to implement, and it targets exactly the failure the customer experiences as “the bot did not help and now I have to start again”. It is also the strongest correction available for a gate tuned slightly too loose, because it catches the miss on the second attempt rather than the fifth.
Action risk tiers
Escalation is not only about answering. What the agent is allowed to do should be tiered:
| Tier | Examples | Gate |
|---|---|---|
| Read-only | Order status, policy lookup | Standard |
| Reversible write | Update address, resend email | Higher bar, logged |
| Financial or irreversible | Refunds, cancellations, deletions | Human approval, or hard limits |
A refund under $20 with a matching order and a clear reason might be automatic. Above that, a person approves. This is a business decision and should be set by the business, not by whoever wrote the prompt.
Tuning the threshold
The threshold is not a technical constant. It is a business trade, and it needs two numbers.
Cost of a wrong answer. Customer harm, a complaint, a wrong refund, a compliance exposure, rework. Varies enormously by category - a wrong opening-hours answer is cheap, a wrong cancellation-policy answer is not.
Cost of an unnecessary escalation. A human touch you did not need, and a slower response for the customer.
Where a wrong answer is much more expensive than a handover, escalate aggressively. Where they are comparable, tune toward answering. Because this ratio differs by category, the threshold should differ by category too. A single global threshold is the most common design error we see.
The practical method: take the evaluation set, and for each candidate threshold compute what share of cases would be answered correctly, answered wrongly, and escalated. Plot the three. You are looking for the point where the wrong-answer rate drops below your tolerance, and then you accept whatever escalation rate that implies rather than choosing the escalation rate first.
Teams that pick the escalation rate first - “we want 70% deflection” - end up with a system that answers things it should not, because the number came from a business plan rather than from measured risk.
What good escalation looks like to the customer
Design the handover, not just the decision. The gate can be perfectly tuned and still produce a bad experience.
- No restart. The person receives the full transcript and a written summary of what the customer wants and what has been tried.
- No pretence. Do not have the agent claim to be human. It costs trust when discovered, and it will be discovered.
- Set the expectation. Say what happens next and roughly when.
- Preserve context on both sides. The customer should not have to repeat their order number, and the agent should see the resolution so the next contact is informed.
A well-handled escalation is a good outcome, not a failure. Systems that treat escalation as a defeat metric get tuned toward answering things they should not.
What we set by default
Our default configuration on a new support agent gates on retrieval quality first, applies topic rules as hard constraints, escalates all repeat contacts within 24 hours, and starts deliberately conservative - escalating more than the eventual target - for the first two weeks.
The reason for starting conservative is that the first fortnight of production traffic tells you things the evaluation set did not, and it is much easier to loosen a gate against real data than to repair the trust damage from a system that was too confident on day one. The escalation rate in week one is not the number to judge it by.
We also treat “answered from a weak source” as a distinct failure category in evaluation reports, separate from “answered incorrectly”. They have different fixes: one is a retrieval or content problem, the other is a reasoning problem, and averaging them together hides which one you have.
When a confidence gate is the wrong tool
When the real problem is missing content. If the agent escalates 60% of a category because nothing in your help centre covers it, tuning the gate is pointless. Write the article.
When the process needs a human by design. Some workflows should not be automated at the decision point regardless of how good the model is. Encode that as a rule and stop tuning.
When there is nobody to escalate to. A gate that routes to an unstaffed queue is worse than no gate, because the customer waits instead of getting a partial answer and a clear next step.
Very low volume. Below a few hundred conversations a month, there is not enough data to tune anything meaningfully. Set conservative rules and leave them.
Frequently asked questions
What escalation rate should we aim for?
Do not aim for one. Set the wrong-answer tolerance by category, and accept the escalation rate that produces. For most support agents that lands somewhere between 30% and 60% initially, falling as content improves.
Can we use a second model to check the first?
Yes, and as a check on whether the answer is supported by the retrieved source it works reasonably well. As a general correctness check it is weaker than it appears. Calibrate it against human judgement before trusting it.
Should the customer be able to request a human at any point?
Yes, always, and prominently. Hiding the escape hatch to protect a deflection metric produces exactly the behaviour where customers learn to type nonsense to defeat the bot.
How do we handle escalation outside business hours?
Decide deliberately. Options are to widen what the agent handles overnight, to collect and queue with a clear expectation set, or to say plainly that a person will respond in the morning. The bad option is silently escalating into a void.
Does a higher threshold hurt the business case?
Short term, yes. Long term, a system that answers wrongly generates repeat contacts, complaints and distrust, and those are also costs. Measure both sides before tightening the business case at the expense of the gate.
Next step
If your agent is escalating too much, too little, or unpredictably, the AI evaluation and QA engagement measures the gate against your real cases and tunes it per category against measured cost.
Related: How to measure whether an AI system works · Latency budgets for conversational AI · Building an eval set from real tickets · AI customer support agents