What causes latency in an AI voice agent?

Turn latency is a chain, and the model is only one link. The caller's experienced delay is the sum of endpointing (how long the system waits to decide they finished speaking), transcription, model reasoning, any tool calls to external systems like a calendar or CRM, speech generation, and the telephony path itself. In production the two biggest and most fixable items are almost always endpointing settings that are too patient for short answers, and blocking tool calls that make the caller wait in silence while the agent writes to a CRM or fires an automation. Optimizing the model while those two are untuned is wasted effort.

How do you reduce AI voice agent response time?

Work down five things in order. Tune endpointing per moment instead of globally, so short answers get short waits and phone numbers get patience. Move every action that the agent's next sentence does not depend on, such as CRM writes, tagging, SMS and summaries, out of the call and into the automation layer behind it. Cover the one wait worth taking, checking real calendar availability, with speech that was going to happen anyway rather than silence. Cut the agent's own replies down, starting with the greeting, because generated speech plays at human speed. Finally, shrink the prompt and tool list each node carries so the model is doing a smaller job per turn.

What is a good endpointing setting for a voice agent?

There is no single good value, which is why most agents feel wrong. Endpointing should vary by what the agent just asked. Short confirmations and yes or no answers deserve a short wait, since lingering there is what makes an agent feel sluggish across a whole call. Open questions like why someone is calling deserve a longer wait, because people think out loud and pause mid sentence. Anything recited in chunks, phone numbers and addresses in particular, deserves the most patience of all, since an agent that jumps in after the first chunk captures a wrong callback number and loses the lead regardless of how fast the call felt.

Why Your AI Voice Agent Feels Slow (It Is Almost Never the Model)

TL;DR: When a client tells me their AI voice agent "feels slow", the model is almost never the problem. The delay lives in the gaps: how long the agent waits before deciding you finished talking, how much work it tries to do while the caller sits in silence, and how long its own sentences are. Latency on a phone call is a budget you spend across a chain, not a number you optimize in one place. Here is how I break that budget down on real production agents, and the rules I follow so a caller never hears the system thinking.

A voice agent that answers correctly two seconds late is worse than one that answers adequately right away. Callers do not experience your architecture. They experience a pause, and a pause on a phone call means one of three things to a human being: the line dropped, the person is confused, or nobody is listening. All three make people talk over the agent, repeat themselves, or hang up.

I have spent the last stretch of my work building production voice and chat agents on Retell for clients in the US and UK, wired into n8n, GoHighLevel and Twilio. Before that I spent years in Unity and AR, where a late frame was a safety problem rather than a conversion problem. That earlier work is where I learned to treat latency as a property of the whole chain, and it is the single most transferable thing I brought into voice AI.

The number that matters is turn-taking, not model speed

The metric people quote is time to first token, or the vendor's published response time. The metric callers actually feel is turn latency: the gap between the moment they stop speaking and the moment they hear the agent start speaking.

That gap is the sum of a chain, and the model is only one link in it:

Endpointing. The system waits to be confident the caller is done talking. This is a deliberate wait, and it is frequently the largest single item in the budget.
Transcription. Speech becomes text, usually streaming, so much of this overlaps with the caller still talking.
Reasoning. The model reads the conversation state and decides what to say, plus whether to call a tool.
Tool calls. Anything the agent has to ask an external system before it can answer, such as a calendar lookup or a CRM read.
Speech generation. The reply becomes audio, and audio takes real time to play regardless of how fast it was generated.
Network and telephony. The call path itself adds delay you do not control.

Two things follow from writing the chain out. First, shaving 200 milliseconds off the model while leaving a two second endpointing setting untouched is wasted effort. Second, steps 4 and 5 are where most badly behaved agents actually lose the caller, and both are within your control as the person who built the thing.

Endpointing: the tradeoff nobody tunes

Endpointing is the agent deciding you are finished. Set the wait too short and the agent interrupts people mid-sentence, which is the rudest failure mode a phone system has. Set it too long and every single turn carries a dead pause, which makes the agent feel slow even when everything downstream is fast.

The mistake is treating it as one global setting. Different moments in a call deserve different patience:

Short answers get short waits. "Yes", a phone number, a confirmation. The caller is done, and lingering here is what makes an agent feel sluggish across an entire call.
Open questions get longer waits. When you ask why someone is calling, they think out loud, pause mid sentence, and resume. Cutting them off here costs you the actual reason for the call.
Anything read out digit by digit gets the most patience. People recite phone numbers and addresses in chunks with real gaps between them. An agent that jumps in after the first chunk will get the number wrong, and a wrong callback number is a lost lead no matter how fast the call felt.

This maps cleanly onto how I already structure agents. I wrote about sizing conversation flow nodes and where I turn flex mode up or down, and endpointing follows the same shape: be loose where the caller is exploring, tight where the caller is confirming.

Rule one: the caller never waits on a slow system

This is the rule that fixes most "slow agent" complaints I get handed.

When an agent is wired into a real business, it touches other systems: a CRM, a calendar, a messaging platform, an automation layer. The instinct is to have the agent do all of it inline, mid conversation, while the caller listens to nothing. That is how you get four seconds of silence after someone says "yes, book me in".

The fix is to split the work by whether the caller's next sentence depends on it:

Blocking work is anything the agent must know before it can speak truthfully. Real availability before it offers a slot. Whether an account exists before it claims one does. This is a short list, and it should stay short.
Non-blocking work is everything else. Creating the CRM contact, tagging the lead, firing the confirmation SMS, writing the call summary, kicking off follow up. None of that has to happen before the agent's next sentence, so none of it belongs in the caller's waiting time.

In practice this means the agent hands off to the automation layer and keeps talking. I described this shape in detail in how I connect Retell to n8n: the agent fires a webhook, n8n does the slow work, and the call carries on. The dashboard and CRM layer behind the agent can catch up a few seconds later, because no human is standing at the CRM watching for the row to appear.

The one wait worth taking, and how to cover it

Calendar availability is the honest exception. If the agent offers a slot it has not verified, you have traded a two second pause for a double booking, which is a far more expensive problem.

So take the wait, but never take it in silence. Speak first, then look up. "Let me check what we have this week" costs about a second and a half of speech, which is roughly the time the lookup needs, and the caller experiences zero dead air because a human was talking the whole time. That is not a trick. It is exactly what a receptionist does while their booking screen loads.

The rule generalizes: if you cannot remove a wait, cover it with speech that would have happened anyway. What you must not do is fill the gap with fake filler that promises progress the system is not making, because callers notice an agent that says "one moment" three times in a row.

Rule two: the agent's own sentences are part of the budget

This one surprises people. Generated speech plays at human speed. A four sentence answer takes four sentences worth of seconds no matter how fast the model produced it. If your agent opens with a paragraph, you have spent more of the caller's patience on your greeting than on any technical delay in the stack.

What I do about it:

Cap the reply length in the prompt and mean it. One or two sentences, then a question. Long, thorough answers read beautifully in testing and feel interminable on a phone.
Front load the useful part. Answer, then explain if asked. A caller who has their answer will interrupt the rest, and that is a good outcome.
Never read back more than necessary. Confirm the details that would be expensive to get wrong, and skip the rest.
Keep the greeting to one line. The greeting is the first impression of speed for every single caller.

Rule three: keep the model's job small

Prompt size and tool count both cost time, and both grow quietly. Every rule someone adds after a bad call, every edge case pasted into the system prompt, every tool bolted on for a feature used twice a month, adds work to every turn of every call forever.

The structural fix is the same one that makes agents more reliable in general: break a long single prompt into a flow where each node carries only the context and tools it actually needs. The node collecting a phone number does not need the refund policy in its prompt. That is a correctness win first and a latency win second, which is my favorite kind of change. If you are choosing a platform for this, I compared how Retell, Vapi and Bland handle structure and orchestration separately.

How I actually test this

Average latency is a comforting and largely useless number. Callers do not remember the average turn. They remember the one four second gap in the middle of giving their address.

So I test for the worst case, not the mean:

Listen to whole recordings, not dashboards. Latency problems are obvious in twenty seconds of audio and invisible in a summary metric.
Test the paths that touch other systems. The booking turn, the lookup turn, the handoff turn. That is where the seconds hide.
Test messy callers. Someone who volunteers their number before being asked, someone who pauses mid sentence, someone with background noise. These are the turns where endpointing settings reveal themselves.
Test at the times the business is actually busy, because third party systems slow down under load and your agent inherits that.

This is where the build and test pipeline I put together at Fortell AI earns its keep. I wrote about using Claude Code to scaffold agents and Comet browser automation to run them against messy call scenarios, and latency regressions are exactly the class of problem that only shows up when you can replay awkward calls cheaply and often.

Why I think about this the way I do

In my Unity and AR years, including the public safety XR work I did at ARCortex, latency was not a conversion metric. If an overlay lagged the operator's head, the illusion broke and the person stopped trusting the headset entirely. Once a user decides a system is lying to them, no amount of accuracy wins them back.

The architecture that solved it there is the same one that solves it on a phone call: decouple what the user feels from what the system does. Acknowledge immediately. Reconcile in the background. Never let slow work freeze the surface a person is depending on. A firefighter looking at a stale marker and a customer listening to three seconds of silence are having the same experience, which is the system failing to prove it is still there.

The short version

If an agent feels slow, work down this list in order:

Tune endpointing per moment instead of globally, tight on short answers, patient on open questions and digits.
Move everything that is not needed for the next sentence out of the call and into the automation layer.
Cover the one unavoidable wait, availability, with speech that was going to happen anyway.
Cut the agent's own sentences down, starting with the greeting.
Shrink prompts and tool lists per node.
Test the worst turn on messy calls, not the average turn on clean ones.

Nothing on that list is exotic, and none of it requires a faster model. Speed on a phone call is a design decision about where the work happens, made repeatedly, and it is usually the difference between an agent a business trusts with its phone and one that quietly costs it leads.

I build production voice and chat agents on Retell, wired into n8n, GoHighLevel and Twilio, for teams that need the phone answered properly and answered fast. If that is you, my inbox is open: more about my background here, or book a call.