TL;DR: An AI voice agent is a chain of rented services: a phone number and carrier, a telephony layer, the voice platform, a speech-to-text model, an LLM, a text-to-speech voice, and your own tools behind it. Any link can fail, and when one does, the caller does not see an error page. They hear silence, a dead line, or an agent that suddenly cannot book anything. The fix is deciding in advance what every failure sounds like: a fallback route that ends at a human or a voicemail, an agent that degrades to taking a message when its tools are down, a scheduled test call so you find out before the owner does, and a short written plan for who flips what. This is the outage plan I put in place on my Retell AI, Twilio, n8n and GoHighLevel builds before launch, not after the first bad morning.
A business that moves its phones onto an AI agent has made the agent part of the front door, and a front door needs a plan for the day the lock jams.
I already wrote about the failures that happen after a good call, when the booking or CRM write silently never happens, in why voice agent automations fail after the call. This post is about the front half: the call itself not going right because something underneath it is down or degraded.
Map the chain before you plan for it
When I scope an agent, I write out every service a single call touches, in order. For a typical build it looks like this:
- The business's existing number and carrier, often forwarding to a new number.
- The telephony layer, usually Twilio or the voice platform's own numbers.
- The voice platform, Retell AI in most of my work, which runs the conversation.
- Speech-to-text, the LLM and the voice, which the platform orchestrates but which may come from different providers.
- The tool layer: n8n workflows, a calendar, GoHighLevel, a practice management system.
- The human fallback: a transfer number, a ring group, a voicemail box.
"The AI is down" is not one scenario. It is five or six, and they need different answers.
The failure shapes, from the caller's side
Dead air or a dropped call
The worst one. The number rings and nobody picks up, or it picks up into silence and then drops. This usually means the problem is at the top of the chain: forwarding broke, the number got detached from the agent, the telephony account has a billing or configuration issue, or the voice platform is not accepting calls.
The caller assumes the business is closed and calls the next result on Google.
An agent that answers but cannot do anything
The conversation layer is fine, but a tool is failing. The calendar lookup times out, the CRM rejects the write, the n8n workflow is deactivated. The agent greets the caller, gets their name and need, and then cannot check availability or book.
Pickup rate looks healthy, which makes this one sneaky. An agent not designed for it stalls in silence, invents a time slot, or ends the call vaguely. I cover how to design tool calls with real timeouts in voice agent function calling.
An agent that sounds wrong
Higher latency, a glitching voice, transcription that mishears everything. Usually a speech or model provider is degraded rather than down, and people hang up mid-sentence. The latency side of this is its own topic, which I wrote up in the voice agent latency budget.
Rule one: the number always ends somewhere
The single most important decision is what happens to a call when the agent cannot take it at all. My rule is that the phone number must always end at something that works without the AI: a human, a ring group, or at worst a voicemail box someone actually checks.
How that is wired depends on where the number lives:
- If the number sits in Twilio and points at the agent, Twilio lets you set a fallback handler that runs when the primary one fails. I point that at a simple flow that forwards to the business's backup line or records a voicemail and emails it. It turns the worst outage into a slightly worse phone experience instead of a dead line.
- If the business forwards its existing line to the agent, the carrier's forwarding setup matters. Overflow forwarding, where the front desk rings first and the agent catches what is missed, has a natural fallback built in, because staff still get the call first. All-calls forwarding has no fallback unless you build one. I went through these choices in detail in setting up the phone number behind an AI receptionist.
- Either way, I keep the original number in the client's hands. If they can log into their carrier and turn forwarding off in two minutes, the worst case has a known, fast, manual way out that does not need me.
The test is simple: if the voice platform vanished right now, what would a caller hear? If the answer is "nothing", that is the first thing to fix, before any prompt tuning.
Rule two: the agent degrades to a message taker
When the conversation layer works but the tools do not, the agent should not pretend. It should fall back to the one thing it can always do: take a good message and promise a callback.
In practice that means three things in the design:
- Every tool call has a timeout and a spoken fallback. If the availability lookup does not return in a few seconds, the agent says something like "I'm having trouble pulling up the calendar right now. Let me take your details and someone will call you back to confirm a time." It never invents a slot.
- The message is captured somewhere independent of the failing tool. If the CRM is down, writing the message to the CRM does not help, so the details also land in a team notification or a stored record the post-call workflow can replay later. The structured fields I capture are the same ones described in capturing caller details.
- The transfer path is the escape hatch. If the caller wants a person and a transfer number exists, the agent offers it. A tool outage is exactly when a human handoff earns its keep. The mechanics are in voice agent human handoff and transfers.
A caller told "the team will call you within the hour" mostly shrugs. A caller who hears eight seconds of silence hangs up.
Rule three: find out before the owner does
The embarrassing outage is the one the clinic manager reports to you at 10am. Monitoring is how you avoid that call.
A scheduled test call
The most useful check I run is a scheduled real call to the agent's number from outside the stack. It waits for the greeting, asks one scripted question, and checks that the post-call event arrived. Any failure alerts someone.
This catches what dashboards miss at the top of the chain: a changed forwarding rule, a number detached during an edit, a billing hold on the telephony account. None of those produce an error in the voice platform, because from its point of view no call ever arrived.
Watch for absence, not just errors
Errors are easy to alert on. Silence is not. I track the normal call volume per hour for each business and alert when it drops to zero during a period that usually has calls. A vet clinic that normally gets a dozen calls on a weekday morning and has received none since 8am has a problem, even if no system has thrown anything.
Pickup rate is the other number I watch closely, and I explain why in voice agent metrics that matter.
Subscribe to your vendors' status pages
Retell, Twilio, the LLM providers and most automation tools publish status pages. Subscribing turns "is it us or them?" into a glance, though it never tells you your own forwarding rule broke.
Rule four: know what you will switch, and who switches it
Much failover should be a deliberate, practised manual step, because automatic switching between complex systems can fail in more confusing ways than the outage itself.
For each agent I write a one page runbook with the client:
- Who gets alerted, me and a named person at the business.
- The kill switch: how to send calls straight to staff or voicemail, with screenshots of the carrier or Twilio setting, so the client can do it without me.
- What gets said to callers, if the outage runs long. Sometimes the right move is a short recorded message, sometimes it is simply staff answering again for a morning.
- The model fallback, if there is one. Switching to a second LLM only helps if it has already been tested against the same prompt. An untested model is a different agent. I compare model choices in choosing the LLM for a voice agent, and It goes through the same regression testing as any other change.
- The cleanup: after the outage, who works through the messages that piled up and calls people back.
This runbook is part of the handover, not an afterthought. A client who owns their agent should also own its emergency procedure, which is a theme from handing a voice agent over to a client.
A pre-launch outage checklist
I do not try to build an agent that never goes down; every service has bad days. The honest target is that no outage produces dead air, loses a caller's details, or lasts longer than it takes a named human to notice and flip a switch. Before an agent takes real calls, I check that:
- Every service a call touches is written down, in order.
- The number has a fallback route that ends at a human or a checked voicemail.
- The client can turn forwarding off themselves in two minutes.
- Every tool call has a timeout and a spoken fallback, and the agent never invents availability.
- Caller details are captured somewhere independent of the CRM.
- A scheduled test call runs against the real number and alerts on failure.
- Zero-volume alerts exist for hours that normally have calls.
- Vendor status pages are subscribed to.
- A one page runbook names who is alerted, the kill switch, and who handles the backlog.
None of this shows up in a demo. It is the difference between the day the phones were a bit odd and the reason the owner turns the agent off.
I build production AI voice and chat agents on Retell AI, with Twilio, n8n and GoHighLevel behind them, and I design the failure paths before the happy path goes live. If you run an agent with no outage plan, or want one built properly from the start, more about my background here, or book a call.