Do AI voice agents struggle with strong accents?

Less than people expect on conversation, and more than people expect on data. Ordinary sentences carry enough redundancy that a mishearing usually survives, so a caller with a heavy regional accent asking about opening hours is understood fine. Identifiers are where it breaks: postcodes, registration numbers, surnames and email addresses have no surrounding context to repair them. The fix is not a different speech model, it is confirmation strategy. Read identifiers back, validate them against the shape the business expects, and offer keypad entry after a couple of failed attempts.

Should one voice agent handle multiple languages, or should each language get its own?

My default is one number and one agent per language. A language is not a setting, it carries its own voice, its own authored knowledge base, its own transfer target and its own post-call messages, so separating them keeps each agent as simple as a normal single-language build and keeps testing sane. A single agent that detects and switches language mid call is worth the complexity in one situation only: when individual callers genuinely switch language inside a single conversation, which happens in real bilingual communities and rarely elsewhere.

What is the most common mistake when adding a second language to a phone agent?

Stopping at the agent. The call goes beautifully in Spanish, then the transfer lands on a reception team that speaks only English, or the confirmation text goes out in English. Language support has to reach the humans behind the handoff and every post-call automation, or it produces a worse experience than not offering the language. If the volume does not justify that, the honest build is an agent that recognises the language, says one clear scripted line in it offering a callback, and captures the number.

Multilingual Voice Agents: One Number Per Language Beats One Clever Agent

TL;DR: Accents and languages are two different problems that get sold as one. Accent variation breaks identifiers, not conversation, and you fix it at the capture layer with confirmation strategy rather than by swapping models. Language is an architecture decision, and my default is one number per language with a dedicated agent behind each, because a language is not a setting, it is a voice, a knowledge base, a set of business hours, a CRM field and a human on the other end of the transfer. Build one agent that switches languages mid call only when callers genuinely switch mid call, and know that you have just doubled everything downstream when you do.

Nearly every voice agent I have shipped runs in English. That includes the UK builds at Fortell AI for hospitals, vet clinics, estate agents and car garages, and the US work on CallSetter AI at Tested Media. So the interesting pressure never came from callers switching language mid sentence. It came from the fact that "English" on a real phone line covers an enormous spread, and from clients asking, usually a month in, whether we could add Spanish or Urdu or Polish "as an option."

Those two questions have completely different answers, and merging them is how projects end up with one bloated agent that is mediocre in three languages.

Accent is a capture problem, not a language problem

Here is the part that surprises people: heavy accents rarely break the conversation. They break the identifiers inside it.

Modern speech models handle conversational speech across accents far better than the demo-driven fear suggests. A caller with a thick Glaswegian accent asking whether you are open on Saturday gets understood, because the sentence has redundancy. Every word around "Saturday" props it up. If the model mishears one token, the meaning survives.

Now ask the same caller for their postcode, their registration number or the spelling of their surname. All the redundancy is gone. Nothing in the surrounding audio tells the model what a postcode should have been. That is the exact failure shape I wrote about in capturing caller details on a live phone line, and it is the same thing I hit years earlier on the transcription side with lecture audio versus phone audio: course jargon and proper nouns were the tokens that broke, because context could not repair them.

So when a client says "the agent struggles with our customers' accents," my first move is never to change the model. It is to look at where in the call the failures cluster. They cluster on data capture, every time.

What actually helps:

Confirm identifiers, do not confirm sentences. Read back the registration, the postcode, the phone number. Do not read back the reason for the call.
Constrain the field where the business lets you. A UK vehicle registration, a postcode, an appointment reference and a phone number all have shape. Validating against the shape catches more than any accent tuning does.
Offer the keypad as a fallback for numbers. DTMF entry on a third failed attempt is not an admission of defeat, it is the fastest path to a correct phone number in a noisy car.
Fix the audio path before blaming the model. A caller on speakerphone in a workshop is an audio problem. No prompt change fixes a bad input.

That is most of what accent robustness actually is in production. Unglamorous, and it works.

Language is an architecture decision

Language is different. A language is not a toggle on your agent config. Look at everything hanging off it:

The voice has to change, and the voice is a brand decision, not a dropdown. I wrote about how much that choice carries in choosing a voice for your AI phone agent. The good voice you picked in English does not have an equally good twin in your second language, and a native speaker will hear the difference in the first sentence.

The knowledge base has to change. Your pricing, policies and FAQs written in English do not become a Spanish knowledge base because the model can translate. Translated-on-the-fly policy answers are where agents invent things, and the client cannot proofread a language they do not read. Everything I said about what belongs in a knowledge base and what belongs in the prompt applies twice over here.

The transfer target has to change. This is the one that kills naive builds. If your agent handles Spanish beautifully and then transfers to a reception team that speaks only English, you have built an elaborate way to waste a caller's time. Language support has to extend past the agent to the humans behind the handoff, or you should not offer the language at all.

The post-call layer has to change. Summaries, SMS follow-ups, CRM notes and email confirmations all have a language now. I have seen a pipeline produce a perfect Spanish call and then text the lead a confirmation in English, which is exactly the kind of silent mismatch that belongs in the list of post-call automation failures.

None of that is hard. It is just four times the work people budget for, and the reason "just add a language" is never a small ticket.

Three architectures, and the one I default to

One number per language, one agent per language. This is my default. Callers get a dedicated line, the agent is configured natively for that language end to end, the voice is chosen properly, the knowledge base is authored not translated, and the transfer routes to people who speak it. Testing is straightforward because each agent is a normal single-language agent. The cost is that you have to publish a second number and tell people about it, which for most local businesses is a footer line on a website.

One number, language chosen at the top, then a dedicated agent. The caller hears a short bilingual greeting, states or selects a language, and gets routed into a language-specific agent. Slightly worse first impression, much better than a menu tree, and you keep the clean separation behind it. This is the compromise I reach for when the client only has one number they can realistically promote.

One multilingual agent that detects and switches mid call. Genuinely useful in exactly one situation: when individual callers switch language inside a single conversation, which happens in real bilingual communities and almost nowhere else. The cost is that every part of the system now has to be language-aware at runtime, including the parts that fire after the call, and your test matrix doubles for every language you add.

This is really the same question I answered in how many voice agents do you actually need: split when the jobs are genuinely different, keep one when they are not. Two languages are two different jobs far more often than they are one.

What breaks when an agent switches languages mid call

If you do build the switching version, these are the failures I would test for first, because they are the ones that hide:

The voice changes identity. The agent sounds like one person in English and a different person in Spanish. Callers notice, and it reads as a system, not a receptionist.
The prompt and the speech drift apart. The instructions are English, the conversation is not, and the model starts answering in the language of the prompt under pressure, usually on the longest or most complex turn.
Only half the call switches. The agent switches, the confirmation SMS does not.
Detection triggers on a name. A caller giving a Spanish surname in an otherwise English sentence should not flip the call. Language detection needs hysteresis, not a hair trigger.
The handoff loses the language. Whoever picks up the transfer needs to know which language the call was in before they say hello.

Every one of those is catchable with a scripted call per language and per switch path. It belongs in the same suite as everything else in regression testing your voice agent, because these break silently on prompt edits and nobody calls to complain, they just hang up.

Be honest with the client about the ceiling

The advice I give when a client asks about adding a language is usually this: if the calls in that language are a real share of volume and you have people who can take the transfer, build it properly as its own agent and its own number. If it is two calls a week and nobody in the office speaks it, do not fake it. Have the agent recognise the language, say one clear scripted line in that language offering a callback, and capture the number. That is an honest, working outcome, and it takes an afternoon rather than a month.

The underlying architecture does not change either way. It is the same stack I describe in how I build production AI voice agents on Retell, with n8n carrying every action, and now carrying a language field alongside everything else it passes around.

Serving callers in their own language is one of the highest-trust things a small business can do on the phone. It is worth doing properly, and doing it properly mostly means refusing to treat a language as a checkbox.

I build production AI voice agents and the automation, telephony and CRM layer behind them for founders and operators across the US, UK and Europe, including the UK multi-vertical work I did at Fortell AI. More about my work here, or book a call.