What's a good response latency for a voice agent?

A widely used rule of thumb is to keep the response under about 800 milliseconds — close to the pause length people naturally leave in human conversation. Above roughly a second, callers start to feel the lag, talk over the agent, or assume the line dropped. It's an approximate target, not a hard line, but it's a reasonable bar to hold vendors to.

Why can't the system just take a moment to think for a better answer?

Because a phone call has no visual feedback. On a website, a spinner tells you to wait. On a call, silence reads as a dropped connection or a confused agent, so the caller fills it — by repeating themselves or interrupting. Speed isn't a luxury on voice; it's part of being understood.

Does a faster system mean a less accurate one?

Not necessarily, but there's real engineering tension between the two. The systems that handle it well overlap their stages — they start transcribing and interpreting while the caller is still talking — rather than racing through each step. Ask a vendor how they hit their latency target, not just what the number is.

Sub-Second Latency: Why It Makes or Breaks a Restaurant Voice Agent

Every other quality of a restaurant voice agent — how natural it sounds, how well it understands your menu — sits on top of one unglamorous number: how fast it responds. Get that number wrong and nothing else rescues the call. Get it right and a synthesized voice starts to feel like an actual conversation.

That number is response latency, and the target most serious teams aim for is under roughly 800 milliseconds.

Why 800 milliseconds, specifically

The figure isn't arbitrary. It's tied to how humans take turns when we talk. In natural conversation, the gap between one person finishing and the other starting is remarkably short, often a few hundred milliseconds. We're exquisitely tuned to that rhythm. When a pause runs long, we read meaning into it: hesitation, confusion, or that the other person didn't hear us.

A voice agent that takes a beat and a half to answer trips every one of those instincts. The caller assumes the line dropped and says "hello?" They assume the agent is confused and repeat the order. They talk over the response. None of that is the caller being difficult — it's them reacting normally to an unnatural pause.

Keeping the round trip under about 800 milliseconds keeps the agent inside the window where human turn-taking feels right. It's an approximate target, not a precise threshold, but it's a fair bar to hold a vendor to.

Where the time goes

Remember that a voice agent runs a four-stage loop on every turn: speech recognition, a language model interpreting the words, speech synthesis, and (once the order's done) the POS write. We break that pipeline down in how AI phone ordering actually works.

The catch is that all of those stages share the same sub-second budget. Speech recognition takes some. The language model deciding what the caller meant takes some. Turning the reply back into audio takes some. Add ordinary network round-trips between services and the budget is spent before you've done anything clever.

This is why latency is genuinely hard, not a matter of buying faster servers. Every stage you add, every extra hop, every "let me double-check that against the database" eats into a budget measured in fractions of a second.

The trick: overlap, don't wait

The systems that hit the target don't do it by rushing each step. They do it by refusing to run the steps strictly one after another.

A well-built agent starts transcribing the first words while the caller is still speaking, and starts interpreting the order before the sentence is even finished — a technique built on streaming rather than waiting for a complete utterance. By the time the caller stops talking, much of the work is already done, so the response can begin almost immediately.

This overlap is also what makes graceful interruptions possible: because the system is always listening, it can stop talking the instant the caller cuts in. Latency and interruption handling are really the same engineering problem viewed from two angles.

The added difficulty of a noisy restaurant

Latency gets harder in exactly the environment restaurants operate in. A caller phoning from a loud dining room, a car, or a kitchen doorway gives the speech-recognition stage a harder signal to work with. Cleaning up noisy audio takes processing, and processing takes time — the same time the whole loop can't afford to spend. A drive-thru window adds engine noise and a menu speaker into the mix, which is part of why drive-thru and phone ordering are genuinely different technical problems.

How to evaluate it without an engineering degree

You don't need to measure milliseconds. Just call the system yourself and pay attention to how it feels:

Does the reply come back at conversational speed, or is there a beat where you wonder if it heard you?
When you interrupt mid-sentence, does it stop right away, or plow through the rest of its line?
Try it from a noisy spot — a busy room, a car — not just a quiet office. That's where latency and comprehension both get tested.

If the call feels like talking to a person who's paying attention, the latency is where it needs to be. If it feels like a walkie-talkie with a delay, no amount of menu tuning will fix the underlying problem.

The bottom line

Sub-second response time is the foundation the rest of a voice agent stands on. Under roughly 800 milliseconds, the conversation feels natural and callers relax into it. Much slower and they start fighting the system — repeating, interrupting, hanging up. When you evaluate vendors, test the feel of the pace first, because it's the one thing a good menu can't compensate for.