Voice agent latency: why response time matters on the phone, and where it comes from

A landmark study is the one by Tanya Stivers and her colleagues, published in PNAS in 2009. The team analysed video recordings of informal conversations in 10 languages spread across 5 continents, focusing on a moment that is comparable everywhere: the answer to a closed, yes-or-no question. In every language, speakers avoid overlapping and minimise the silence between turns. For each language, the most frequent answer comes between 0 and 200 milliseconds after the end of the question, and the mean delay across the whole dataset is 208 milliseconds.

Differences between languages remain limited: each language’s mean lies within about 250 milliseconds either side of the overall mean. The study also shows that, in every language, actual answers come faster than non-answers, and confirmations come faster than disconfirmations (a statistically significant difference in 7 of the 10 languages). In other words, a delay is not neutral: in the data, it goes hand in hand with non-answers and negative answers.

Two limits to bear in mind: these data concern face-to-face conversations, not phone calls, and French is not one of the 10 languages studied. So they do not set a numerical target for a machine; they are a reminder of the pace your callers are used to.

Even before a voice agent starts thinking, the voice has to travel. Recommendation G.114 of the International Telecommunication Union (ITU), on one-way transmission time, recommends not exceeding 400 milliseconds for general network planning, while stressing that highly interactive exchanges, such as many voice calls, can be affected by much shorter delays.

For a voice agent, the sound crosses the telephone network, reaches servers, then comes back: each leg adds time. The documentation of ElevenLabs, a speech synthesis provider, says as much: end-to-end latency varies with location and with the type of endpoint used, and the response times it publishes differ from one region of the world to another.

An agent cannot answer until it has decided that the sentence is over. At Deepgram, a speech recognition provider, this function is called endpointing: a voice activity detector monitors the audio stream and, by default, finalises the transcription after 10 milliseconds of silence, a setting the developer can lengthen. OpenAI’s Realtime API offers two approaches: a mode that splits the audio on silences, with a configurable silence duration, and a semantic mode that relies on the words spoken to judge whether the user has finished, with an adjustable level of eagerness.

The trade-off never goes away. A short setting makes the agent answer sooner, at the risk of speaking while a caller is searching for words or dictating a phone number. A long setting avoids these interruptions, but every answer starts later. The right setting depends on your callers.

Speech recognition does not necessarily wait for the end of the sentence. Google Cloud’s documentation describes streaming recognition, designed for real time, which can return, if the option is enabled, interim results that are refined as the audio arrives.

Next comes the language model that formulates the answer. Anthropic’s documentation points out that latency depends in particular on the size of the model, the complexity of the prompt and the infrastructure, and that one of the ways it is measured is the time to the first piece of the response. Its levers are simple: choose the right model, shorten the prompt and the response, and stream the response as it is produced rather than waiting for it to be complete.

Finally, speech synthesis turns the text into a voice. Streaming endpoints return the audio progressively, which reduces the delay before the first sound, and ElevenLabs specifies that higher-quality audio formats can increase latency. The delay perceived by the caller therefore depends above all on how quickly each stage passes a first chunk on to the next.

Measure it where the caller feels it: between the end of your sentence and the first audible sound of the answer. A simple, verifiable method is to record your test calls, complying with the applicable rules on informing the other party, then look at the waveform in audio software: silence can be seen and measured.

Vary the situations: a short question, a long question, a pause mid-sentence, a dictated number, a question outside the agent’s scope, a call from a mobile in a noisy place. Look at the slowest cases as closely as the typical ones: a long silence at the wrong moment leaves more of an impression than a decent average.

If a provider puts forward figures, ask how they are measured: on the server side or at the caller’s ear, with what end-of-turn setting, and whether they share the slowest cases or only an average.

Speed is only one criterion. An agent that cuts people off or makes up an answer to avoid a gap can do more damage to the conversation than a slightly longer delay.

Alyas answers the phone 24 hours a day based on the information you give it; if it does not know, it says so and offers to pass the request on, and it places a transcript and a summary of each call in your Alyas dashboard. The 10-day free trial, with no credit card required, lets you run the tests described above yourself.