TL;DR: Most voice agents that "sound bad on real calls" are not failing at language, they are failing at turn taking. Two settings decide almost all of it: endpointing, which is the agent deciding you have finished speaking, and barge-in, which is the agent deciding you have started. Background noise attacks barge-in specifically, because a workshop, a waiting room full of dogs or a car on the motorway all look like a caller who is talking. The fix is not one global setting. It is per agent tuning against the room that agent actually gets called from, shorter agent turns so an interruption costs less, confirmation loops on anything that matters, and an honest handoff when the noise wins.
Every voice agent project I have worked on has hit the same complaint at some point in week one, phrased one of two ways. Either "it keeps talking over me" or "it will not stop talking". Those sound like opposite problems. They are the same problem seen from two sides, and they live in a layer most people never look at, because it is not the prompt and it is not the integrations.
I built production voice agents at Fortell AI for UK hospitals, vet clinics, estate agents and car garages, and voice and chat agents for the CallSetter AI product at Tested Media. The four verticals in that first list sound like a marketing spread. On this particular problem they are four different acoustic environments, and they needed four different configurations of the same agent architecture.
The two settings everyone confuses
Endpointing: has the caller finished?
Endpointing is the agent deciding that a caller's turn is over so it can start responding. It is mostly a timing judgment about silence. I covered what it costs on the clock in the voice agent latency budget, because endpointing is the single largest lever on perceived response time.
Set the wait too short and the agent starts answering while somebody is drawing breath in the middle of their postcode. Set it too long and every turn has a dead pause on the end, which on a phone line reads as a dropped call within about two seconds.
Barge-in: has the caller started?
Barge-in is the other direction. The agent is speaking, the caller begins to speak, and the agent has to decide whether to stop. Humans do this constantly and without thinking. A receptionist who is halfway through "and can I take a number in case we get cut o-" stops instantly the moment you say "oh-seven-nine".
An agent has to be told to do that, and more importantly it has to be told what counts as the caller speaking. That is where the trouble is. Barge-in detection does not hear words. It hears audio energy on the inbound leg and decides whether it is speech. A dog, an impact wrench, a tannoy announcement and a passenger in the next seat are all audio energy on the inbound leg.
Why background noise breaks barge-in and not much else
Speech recognition is far more robust to noise than people expect. Modern models will happily transcribe a caller in a loud room, which is a different problem from the one I wrote about in speech to text for lectures versus phone calls.
Turn taking is not robust in the same way, because it has to make its decision before any transcription exists. It has milliseconds and a waveform. So a noisy call fails in a very specific pattern:
- The agent starts a sentence, the caller's environment makes a noise, the agent stops mid word.
- The agent, now cut off, resumes or rephrases, which produces another partial sentence.
- The caller, hearing fragments, starts talking to try to fix it, which triggers another stop.
Three turns in, the call has degenerated into two parties interrupting each other, and the transcript afterwards looks like the agent had a stroke. Nothing in the prompt is wrong. The agent never got to finish a sentence long enough to be judged on.
The mirror failure is an agent tuned so deaf to barge-in that it finishes a twenty second turn while the caller repeats "hello, hello" into the void. That one is worse commercially, because the caller hangs up and tells the business the robot ignored them.
Four rooms, four configurations
The car garage
Air tools, a radio, a roller door, and the person answering is often holding the phone against a shoulder. This is the loudest environment I have built for. Barge-in has to be deliberately insensitive here, and the agent's turns have to be short enough that insensitivity does not trap the caller. It is also the vertical where the keypad fallback matters most, because a registration plate read out over an impact wrench will not survive on the first attempt.
The vet clinic waiting room
Dogs. Specifically, dogs that bark in bursts, which is close to the worst possible signal for barge-in, because a bark has a speech-like envelope: sudden onset, short, loud. On vet clinic agents I care less about raw sensitivity and more about what happens after a false trigger. The agent should resume the thought rather than restart the sentence, and it should never apologise for being interrupted when it was not.
A car on a motorway
Estate agent and property calls are disproportionately taken by people driving. Road noise is continuous rather than bursty, which paradoxically is easier for barge-in and harder for recognition. The bigger issue here is the speakerphone, and a car cabin's echo, which can feed the agent's own voice back into the inbound leg. An agent that barge-ins on itself will stop speaking every time it says a hard consonant.
A hospital corridor
Tannoys, trolleys, other conversations and often a handset that is passed between people mid call. Hospital work has an extra constraint that sits on top of all this: getting the turn taking wrong here is not a bad customer experience, it is a clinical one, which is part of why agents for hospitals should hand off early rather than push through a bad connection.
What I actually change
Tune per agent, never globally. This is the single most useful habit. The garage agent and the estate agent agent can share a prompt, a knowledge base and every integration behind them, and still need different turn taking. Platforms let you set this per agent for a reason.
Shorten the agent's turns. This is the highest leverage fix available, and it is free. A turn that is one sentence long is barely worth interrupting, and if it does get cut off, the caller already has the part that mattered. I wrote the rules I use for this in writing voice agent prompts for the ear, and the relevant one is to front load the useful clause and put the politeness after it.
Confirm anything you would have to phone back about. Names, numbers, addresses, registrations, dates. Noise does not usually destroy a conversation, it destroys identifiers, and identifiers are the part the business needs. The confirmation patterns and shape validation I use are in capturing caller details.
Give the agent a recovery ladder, not a single line. First unclear turn, ask them to repeat. Second, ask a narrower question or offer the keypad. Third, stop trying and offer a callback or a human. Three strikes is roughly where a real person would give up too.
Do not let it apologise for phantom interruptions. If the agent stops because a door slammed and then says "sorry, go ahead", the caller has to work out what happened. Silence and a clean resume is better.
Cover waits with speech, not silence. If the agent has gone quiet because a calendar lookup is running, the caller will assume the line dropped and start talking, which triggers barge-in, which now collides with the response arriving. One short line before a known wait removes a whole class of collision.
The thing you cannot fix with settings
Sometimes the connection is simply too poor, and every extra turn you spend trying makes the caller angrier. The correct behaviour then is to stop being clever: acknowledge the line is bad, capture a number if you can get one cleanly, and either transfer or promise a callback. I would rather an agent hand off early than grind a caller through six failed attempts at a postcode.
This is a product decision more than an engineering one, and it is worth making explicitly with the client before go-live rather than discovering it in a recording afterwards.
How to test it before a caller finds it
You cannot evaluate turn taking by reading transcripts, which is the trap, because transcripts are what everyone has open. A transcript of a badly interrupted call and a good one can look nearly identical in text.
So test by ear, on a real line, in the real room. Call the vet agent from a waiting room. Call the garage agent from the workshop floor with the radio on. Call the property agent from a moving car on speakerphone. Then, once you know the settings hold, fold those calls into the scripted set you re-run before every change, which is the practice I described in regression testing a voice agent. Turn taking regressions are sneaky, because a prompt edit that lengthens the greeting by one sentence can undo a week of tuning.
Most of this surfaces anyway in the first two weeks live, which is the honest place to expect it. It is much cheaper to find it in a workshop with a test number than on the main line.
The short version
- Endpointing decides when the caller stopped. Barge-in decides when the caller started. They are different settings and they fail in opposite directions.
- Background noise attacks barge-in, because that decision is made on audio energy before any words exist.
- Tune per agent against the room it is called from, not once across the account.
- Short agent turns are the cheapest fix for both failure modes at once.
- Noise destroys identifiers rather than conversation, so confirm the things that matter and offer the keypad.
- When the line is genuinely bad, hand off. Persistence is not a feature.
- Test by ear from the actual environment, then lock those calls into a regression set.
None of this is the impressive part of building a voice agent, which is exactly why it is usually the part that decides whether the business keeps it. The architecture behind it does not change, and it is the same stack I describe in how I build production voice agents on Retell. This layer just sits underneath all of it, and it is the first thing a real caller in a real room will find.
I build production AI voice agents and the automation, telephony and CRM layer behind them for founders and operators across the US, UK and Europe, including the multi-vertical UK work I did at Fortell AI. More about my work here, or book a call.