TL;DR: An AI voice agent should not try to win an argument with an upset caller, and it should not try to fake empathy at length either. The pattern that works on the phone is short and boring: acknowledge once in plain words, stop talking long enough for the person to finish, say exactly what the agent can do right now, capture the complaint as structured data, and get a human involved quickly when the caller wants one. The agent must never promise refunds or compensation, never blame the caller or the staff, and never pretend to be a person. I build production voice agents on Retell AI for hospitals, vet clinics, car garages and real estate teams, and the upset caller is the call I design for first, because it is the one the owner hears about.
Voice agent demos are polite people booking appointments. Real phone lines are not. Someone's dog has been vomiting since morning. Someone's car was supposed to be ready yesterday. Someone has called the hospital three times about the same referral. They are angry at the situation, and the agent is the first thing they reach.
A booking that goes slightly wrong gets fixed. A distressed caller who feels brushed off by a robot becomes a review, a complaint to the owner, and often the end of the project.
Upset is not the same as abusive, and neither is an attack
I separate three kinds of difficult call, because each needs different behavior:
- Upset callers have a real problem and are frustrated, worried or impatient. This is most difficult calls, and this post is about them.
- Abusive callers are insulting or threatening. The agent should stay calm, say once that it can help if the conversation continues respectfully, and end the call if it does not.
- Manipulative callers try to talk the agent into something: a discount, someone else's information, an override. That is a security question, and I cover it in voice agent security and prompt injection.
If the prompt treats every raised voice as a threat, the agent gets cold with a worried pet owner. If it treats every pushy request as distress, it gets talked into promises it cannot keep.
What the agent should say: one acknowledgment, then action
Acknowledge once, in plain words
The instinct is to load the prompt with empathy. "I completely understand how frustrating this must be for you, and I'm so sorry for any inconvenience." Spoken by a synthetic voice to someone already annoyed, it sounds like a script, because it is one.
What works is one short acknowledgment that names the actual problem: "That sounds stressful, especially with your car off the road." Then move on. If the agent repeats "I understand your frustration" every turn, callers notice within two exchanges and it makes things worse. I explicitly tell the agent not to repeat the same empathy phrase, and to acknowledge at most once unless the caller raises something new. This is part of the broader point in writing prompts for speech, not text: what sounds warm written down often sounds hollow out loud.
Let them finish
Upset callers talk in long bursts, with pauses in the middle. An agent tuned for snappy booking calls will jump into those pauses and talk over them, and nothing makes a frustrated person angrier than being interrupted by a machine. For lines where difficult calls are common, I make the agent less eager to take its turn and make sure it stops speaking immediately when the caller starts again. The tuning details are in handling interruptions and background noise. A slightly slower agent that lets people finish beats a fast one that cuts them off.
Say what it can do, specifically
After the acknowledgment, the agent should state what happens next in concrete terms. Not "someone will look into this," but "I'm going to pass this to the service desk now with your booking details, and they'll call you back today." If the agent cannot do anything about the problem itself, it should say so honestly and say what it can do: take the details, flag it as urgent, or transfer the call.
Vague reassurance is what upset callers trust least. Keep each turn to one or two short sentences, and ask one question at a time.
What the agent must never do
These are hard rules in every agent I ship, written into the prompt and tested:
- Never promise money. No refunds, discounts, waived fees or compensation. The agent can say the request will be passed to someone who can decide. A voice agent that offers a refund to calm someone down has just created a commitment the business never approved.
- Never assign blame. Not to the caller ("you should have called earlier"), not to the staff ("the mechanic made a mistake"), not to another department. Anything it says can be quoted back to the business.
- Never argue facts. If the caller says they were promised a callback and the records do not show it, the agent should not contradict them. It records the claim and passes it on.
- Never make clinical or technical judgments under pressure. A vet clinic agent does not tell a panicking owner their pet will be fine. A garage agent does not diagnose a breakdown. I wrote about where those lines sit in AI voice agents for vet clinics and AI voice agents for auto repair shops.
- Never pretend to be human. Asked directly, an upset caller deserves a straight answer. Lying about it, when they find out, turns a complaint into a trust problem. My view on disclosure is in call recording consent and AI disclosure.
When to hand the call to a person
The fastest way to calm most upset callers is to let them talk to a human when they ask. I do not design agents that fight to keep the call.
For upset callers specifically, I usually lower the bar for escalation compared with ordinary calls:
- A direct request for a human is honored immediately.
- A second expression of strong frustration, after the agent has already acknowledged and offered a next step, triggers an offer to transfer.
- Anything involving safety, a medical emergency or an animal in distress follows the urgent route, regardless of tone.
The transfer has to carry context and needs a plan for when nobody answers, which I cover in human handoff and transfer. Complaints should get a dedicated route, not the same queue as new bookings.
Capture the complaint as data, not just a transcript
A transcript of a heated eight-minute call is not actionable for a busy front desk. So on complaint calls I have the agent and the post-call analysis pull out a small, fixed set of fields:
- Who: caller name, number, and the matching customer or patient record if one exists.
- About what: the booking, vehicle, pet, property or referral the complaint concerns.
- What happened, in the caller's words, summarized in two or three sentences without editorializing.
- What they want: a callback, a reschedule, a manager, a refund request, information.
- What the agent already told them, especially any next step it promised.
- Urgency, as a simple level the business has defined in advance.
The field people forget is the promise. If the agent said "the service desk will call you today," that promise has to land in front of the service desk, or the business has broken a commitment it did not know it made. I wrote about the failure modes here in post-call automation failures, and about writing summaries that staff actually read in call summaries staff read.
In my builds the post-call webhook goes to n8n, which routes complaint calls to a different place than routine ones: a flagged task in the CRM, a message to a manager, or a dedicated channel. On the CallSetter AI work at Tested Media, GoHighLevel was the CRM, so complaint calls became tagged contacts and tasks the team could filter, rather than notes buried in a call log.
Use memory carefully with repeat callers
A caller ringing for the third time about the same issue is the most frustrated caller you will get. Greeting them like a stranger is the worst move. If the CRM shows an open complaint for that number, the agent should know it and say so: "I can see you called earlier about the referral. I'm sorry it's still not sorted." That one sentence does more than any empathy script.
It also needs restraint: no reading back private details to someone who has only given a phone number, and no claiming progress it cannot see. I covered the wiring in returning callers and CRM memory.
Test the hard calls on purpose
Upset callers rarely appear in a client's test calls, because testers are calm. So I write them into the test set deliberately: a caller who interrupts constantly, a caller who demands a refund, a caller who claims they were promised something, a caller who swears, a caller who says "just put me through to a human" in the first sentence. Each one has an expected outcome, and every change to the prompt gets run against them again.
This is the same discipline as my regression testing for voice agents and the automated pipeline I used at Fortell AI, where I built and tested agents with Claude Code and Comet browser automation. Simulated difficult callers catch obvious regressions, like a prompt change that makes the agent apologize every turn.
After launch I watch sentiment and escalations alongside booking metrics. A rise in upset transfers often signals a change in the business, like a workshop delay or staffing gap, not in the agent. More on what I track in voice agent metrics that matter.
A short checklist
Check that your agent:
- Acknowledges a problem once, specifically, and does not loop on empathy phrases.
- Lets callers finish, even if that makes it a little slower.
- States a concrete next step instead of vague reassurance.
- Never promises money, assigns blame or argues facts.
- Transfers immediately when asked for a human, with context.
- Records complaints as structured fields, including what it promised.
- Routes complaints separately from routine calls.
- Recognizes repeat callers with an open issue.
- Is tested against difficult callers on every change.
None of this is clever. It is about removing what makes a frustrated person more frustrated, and making sure the business can follow through.
I build production AI voice and chat agents on Retell AI, with n8n and GoHighLevel behind them, for teams that need them to hold up on real calls, including the difficult ones. If you want an agent built or a second opinion on one you already run, more about my background here, or book a call.