Latency budgets for conversational AI

Budget from the user’s tolerance backwards, not from the components forwards. Chat tolerates a first token in about one to two seconds if something streams. Voice needs a response starting inside roughly 500-800ms or the silence reads as a dropped call. Then allocate: retrieval, model, and any tool calls each get a slice, and the slice that overruns is almost always a tool call to somebody else’s API.

Latency is treated as an engineering detail until it is a conversion problem. In voice it is a product-defining constraint from the first day.

Key takeaways

Time to first token matters far more than total time, wherever you can stream.
Voice and chat are different products with different budgets. Do not reuse one for the other.
Your own components are rarely the problem; the third-party lookup usually is.
Parallelise retrieval and any lookups that do not depend on each other.
Measure the 95th percentile. The mean hides the experience people complain about.

Who this applies to

You are building or tuning a conversational system - support agent, internal assistant, voice line - and responses feel slow, or you are designing and want a budget before you build.

What people actually tolerate

Context	First response	Total	Notes
Text chat, streaming	1-2s	5-10s acceptable	Streaming changes the perception entirely
Text chat, no streaming	under 3s	under 3s	Every second past this loses people
Voice	500-800ms	continuous	Silence past ~1s reads as a fault
Async, email or ticket	seconds to minutes	minutes	Latency is nearly irrelevant here
Agent with visible progress	2-3s to first update	30s+ acceptable	Only if progress is genuinely shown

Two things drive this table.

Streaming resets the clock. A response that begins in 1.2 seconds and takes eight to complete feels faster than one that appears whole after four. If your interface can stream, the metric to optimise is time to first token and almost nothing else.

Voice has no equivalent. You cannot stream partial thinking into a phone call. Silence is the only signal, and humans read a gap over about a second as something being wrong. This is why voice agents are architecturally different rather than the same agent with speech attached.

Where the time goes

A grounded answer in a typical support agent:

Stage	Typical	Notes
Request handling, auth	10-50ms	Rarely the problem
Embedding the query	20-100ms	One small model call
Vector search	20-150ms	Grows with index size and filters
Reranking	50-300ms	Optional, and often pays for itself
Tool or API lookups	100ms-3s+	The usual culprit
Model call, first token	300ms-2s	Varies by model and prompt length
Model call, completion	1-8s	Scales with output length
Post-processing, validation	10-100ms

The wide row is the third-party lookup. Your retrieval stack is generally predictable; the CRM, the order system, or the shipping API is not, and it is the component you control least.

Typical midpoints added in sequence miss a 1.5 second target by nearly a second. Running the lookups alongside retrieval is the single largest saving, and it costs nothing but the order of two calls.

Budget backwards

Start from the tolerance and allocate down. For a chat agent targeting 1.5s to first token:

100ms request handling and auth
200ms retrieval, embedding and search combined
400ms tool lookups, in parallel with retrieval where possible
700ms model time to first token
100ms buffer

That adds up only because retrieval and lookups overlap. Which is the first thing to fix in most systems: sequential calls where nothing forces the ordering.

If the order lookup does not depend on the retrieved documents, issue both at once. This is unglamorous and it routinely removes 300-500ms.

Buying back time

In rough order of return.

1. Stream. If you are not streaming and your interface can, this is the single largest perceived improvement available, and it changes no output.

2. Parallelise independent calls. Covered above.

3. Set aggressive timeouts on third-party lookups, with a defined fallback. If the shipping API has not answered in 800ms, answer without the tracking detail and say so. A useful answer now beats a complete answer in four seconds, and it also protects you when their service degrades.

4. Retrieve less, rerank better. Fewer chunks means a shorter prompt, which means faster first token as well as lower cost. This is the same change discussed in cutting LLM costs without degrading quality, and latency is the second reason to do it.

5. Route by difficulty. Simple questions to a smaller, faster model. Most traffic does not need the largest model, and smaller models are meaningfully faster to first token.

6. Cache. Repeated identical questions with stable answers should not be recomputed.

7. Trim the prompt. Time to first token scales with input length. A prompt that accumulated forty instructions is slower as well as more expensive.

The thing that ruins voice

Voice pipelines chain speech recognition, the model, and speech synthesis. Each adds latency and the budget is roughly 700ms end to end.

Three implications that change the design.

You cannot afford a slow tool call. A CRM lookup taking 1.5 seconds is not survivable mid-turn. Either pre-fetch likely-needed data when the call connects, or design the conversation so the assistant says something while it waits - which is what a human does naturally and what most voice agents forget.

You cannot afford a long prompt. Time to first token is most of your budget.

Filler speech is a real technique, not a hack. “Let me check that for you” while a lookup runs is how humans handle the same problem. It has to be honest - if you cannot find it, say so.

Cold starts and the first request

A category of latency that does not show in average measurements and generates a specific complaint: the first request after a quiet period is dramatically slower than the rest.

Three usual causes.

Serverless cold starts. A function that has not run recently pays initialisation on the next request. For a support agent with quiet overnight periods, the first customer of the morning gets the worst experience, and they are disproportionately likely to be the one who reports it.

Connection pool warm-up. First requests establish TLS connections to the model provider, the vector store and every API you call. Subsequent requests reuse them.

Cache misses at every layer. Whatever is warm during the day is cold at 7am.

Mitigations, in order of effort: keep a minimum warm instance if your platform supports it; issue a scheduled synthetic request every few minutes to keep connections and instances alive; and establish connection pools at start-up rather than lazily on first use.

The measurement point matters more than the fixes. Report p95 by hour of day, not just overall. A system averaging 1.2 seconds may be averaging 0.9 in the afternoon and 5 seconds at the start of the morning, and the aggregate figure hides the version of the experience that generates the complaints.

Measure the tail

Report the 95th percentile, not the mean.

A mean of 1.4 seconds with a 95th percentile of 6 seconds describes a system where one conversation in twenty is visibly broken. Those are the ones that generate complaints, and they are invisible in the average.

Instrument per stage, not just end to end. When p95 moves, you want to know which component moved rather than starting an investigation. In practice it is nearly always one downstream dependency having a bad afternoon.

Instrument the stages, not the endpoint

A single end-to-end timing tells you a request was slow and nothing about why, which turns every latency question into an investigation.

Record a duration for each stage on every request - retrieval, each tool call by name, the model call split into time-to-first-token and total, and post-processing. Tag them with the model version, so a provider change is visible as a step in the timing series rather than a mystery.

Two things follow immediately. When p95 moves you can see which stage moved, usually within a minute. And you get the distribution per dependency, which is what tells you that the shipping API’s p99 is four seconds and therefore needs a tighter timeout than you set.

The cost is a few milliseconds and some structured logging. It is the cheapest thing on this page and it is what converts latency from an impression into a number with a cause attached.

What we set as defaults

Streaming wherever the channel supports it, retrieval and independent lookups issued in parallel, hard timeouts with defined fallbacks on every third-party call, and p95 latency in the same monthly report as quality and cost.

The default that gets questioned most is the timeout with a degraded answer. It means occasionally answering without a detail that was available a second later, and that feels like accepting a worse answer. The reasoning: a third-party dependency will have a bad day, and on that day a system with no timeout does not become slow, it becomes unavailable. Deciding the fallback in advance turns an outage into a slightly reduced answer.

The related position: we treat voice as a different build rather than a channel added to an existing agent. The 700ms budget makes several patterns that work well in chat - long prompts, sequential lookups, a verification pass - simply unaffordable. Quoting voice as an increment on a chat agent understates it, and we would rather have that conversation at proposal time.

Where we will not optimise: cutting the retrieval-quality gate to save time. Answering faster from a weaker source is not a latency improvement, it is a quality reduction with a latency benefit, and it should be decided as one.

When latency does not matter

Asynchronous channels. Email and ticket responses measured in seconds are fine. Do not spend engineering here.

Batch processing. Throughput is your metric.

Internal tools where users expect a wait, provided progress is shown.

Before quality is acceptable. A fast wrong answer is not an improvement. Fix accuracy first, then latency.

Frequently asked questions

What is a good time to first token?

Under a second is comfortable for chat with streaming. Under 500ms is what voice needs from the model stage alone, which is why voice is architecturally constrained.

Does a bigger model always mean slower?

Generally, particularly to first token, though it varies by provider and load. Measure it on your own prompts rather than assuming from parameter counts.

Should we pre-fetch data before the user asks?

For voice, often yes - fetching the caller’s recent orders when the call connects removes a mid-turn lookup. For chat it is usually unnecessary and it costs money on conversations that never need it.

How do we handle a slow third-party API?

Timeout with a defined fallback, and say what is missing. Never let an upstream dependency set your response time.

Is streaming worth implementing if we already respond in three seconds?

Usually yes. Perceived latency improves substantially even when total time is unchanged, and it is a one-off piece of work.

Next step

If p95 latency is not in your monthly report next to quality and cost, it is probably worse than you think. The AI evaluation and QA engagement instruments latency per stage alongside the quality baseline.