What should you decide before building an AI voice agent?

Four things, in this order. Which specific call is being lost today, because overflow handling, after-hours coverage and speed-to-lead callbacks are three different products. The rough numbers: call volume, average length, share arriving out of hours, and what one booking is worth, which together tell you whether the project pays for itself. What the agent must never do or say, written before the happy path, because those refusals become explicit exits in the flow rather than hopeful lines in a prompt. And who owns access to the phone number, calendar and CRM, since access is nearly always the critical path rather than the engineering.

How do you know if a business is a good fit for an AI voice agent?

Look at the last twenty calls rather than asking what people call about. A voice agent is worth building when a large share of calls are the same small number of things, and painful when the tail is long and nearly every call needs human judgement. If most calls are genuinely judgement calls, the honest recommendation is capture and routing rather than resolution: the agent takes the details, flags the urgency and gets a person involved, which is a much smaller and much more reliable build.

What are the warning signs that a voice agent project will fail?

No single decision maker, because an agent touches phones, calendars, brand voice and staff workflow and four stakeholders with a veto produces a design that changes weekly. A client who wants callers not to know it is not human, which I do not build. No willingness to sit through test calls before launch, which means their customers find the problems instead. Nobody willing to own telling you when prices, hours or locations change, since staleness rather than hallucination is how agents actually go wrong. And an undefined underlying process: if the humans do not agree on how a booking is taken, the agent cannot.

The Questions I Ask Before Building Anyone an AI Voice Agent

TL;DR: Most AI voice agent projects that go badly were lost on the first call, not in the build. The client asked for "an AI receptionist", nobody defined which calls were actually being lost, nobody wrote down what the agent must never do, and nobody checked who owns the calendar. So I stopped taking briefs and started running a scoping conversation with a fixed shape: find the leak, get four numbers, watch how a call is handled today, list the refusals before the happy path, confirm system ownership and access, name an owner for the words, and agree what "working" means before anything is built. The output is one page, and sometimes that page says the honest answer is not a voice agent at all.

I have built voice and chat agents for hospitals, vet practices, estate agents and car garages in the UK, and for US clients running lead capture and appointment setting at volume. The engineering across those is more similar than you would think. The projects that went well and the ones that dragged apart were separated almost entirely by how well the first conversation went.

A brief that says "we want an AI receptionist to answer our phones" is not a specification. It is a symptom description. Here is the conversation I run instead, and what each question is really for.

Start with the leak, not the agent

The first question is never about the agent. It is: which call is currently being lost, and what does losing it cost you?

The answers split into a few recognisable shapes. A garage that stops answering when both bays are full. A clinic where the phone rings out through lunch and after five. An agency where new leads sit for four hours before a human dials them back. A vet practice where the line is fine during the day but nobody covers the out of hours worried-owner call.

Those are four different products. The first needs overflow handling. The second needs after-hours coverage. The third is not really an inbound problem at all, it is an outbound speed-to-lead problem with completely different consent and compliance mechanics. The fourth needs a very careful escalation design and almost no automation.

If I skip this question, I build the generic version of all four, which is the version that does each of them slightly badly.

Get four numbers before anything else

I ask for four things, and I would rather have rough honest numbers than precise invented ones:

Call volume, roughly, per week.
Average call length.
What share arrives outside working hours, or while everyone is busy.
What one booked job or new client is worth.

Numbers one and two tell me the running cost, which I can quote from the actual cost breakdown rather than a vendor page. Number three tells me whether the agent should take every call or only the ones a human did not reach, which changes the whole phone number and forwarding setup. Number four tells me whether this project is worth doing at all.

That last one saves everybody time. If a business takes thirty calls a week and a booking is worth forty pounds, a bespoke build does not pay for itself and I say so.

Watch how one call is handled today

Then I ask someone to walk me through a real call, out loud, in the order it actually happens. Not the policy. The behaviour.

This is the single highest-value part of the call, and it is where the real spec lives. What I am listening for is: which questions get asked first, which systems get opened, where the answer is written down, what gets promised to the caller, and what the person does when they do not know something.

Nearly every important design decision falls out of that walkthrough. If the receptionist checks a paper diary and then a software calendar, I have a two-source-of-truth problem to solve before I touch the booking layer. If they say "I just know who that is from the number", caller lookup matters more than caller capture. If they say "I take a message and someone calls back", the agent's job is capture and routing, not resolution, which is a smaller and far more reliable build.

Ask for the last twenty calls, not the imagined ones

If I ask a client what people call about, I get a tidy list of five things. If I ask them to look at their last twenty calls, I get a messier and far more useful one, usually including two or three categories nobody mentioned.

The distribution is what matters. Voice agents are worth building when a large share of calls are the same small number of things, and painful when every call is a judgement call.

That inventory also becomes the test set later, and it decides what goes into the agent's knowledge layer versus a live lookup. Anything asked constantly and stable goes in the prompt, the long tail goes in the knowledge base as short spoken answers, and anything that can change today gets looked up live or does not get answered.

Write the refusals before the happy path

Then I ask the question clients find strange: what should this thing never say or do, even if it could?

I ask it early and deliberately, because the answer is the safety boundary and because it is easier to get an honest one before anyone is excited about the demo. In medical work it covers anything resembling triage or advice. In vet work it is the sick-animal call, which goes to a human immediately, every time. In property it is steering, pricing negotiation and anything legal or financing related. In a garage it is diagnosing a fault over the phone.

Every one of those becomes an explicit exit in the flow rather than a line in a prompt, which is why I write the handoff rules before the happy path and scope the tools so the model cannot do the thing it is told not to do.

The related question, asked the same way: what would embarrass you if a caller heard it? That one surfaces tone constraints that never make it into a written brief.

Confirm who owns the systems, and who can grant access

This is the boring question that decides the timeline. For each system in the loop I ask who owns it and who can grant access today: the phone number and where it is registered, the calendar, the CRM, the website, the domain records if texts or email are involved.

The failure I have seen most often is not technical. It is that the phone number sits with a provider only the previous marketing agency can log into, or the calendar everyone actually uses is a personal one, or nobody in the room can approve the A2P registration the SMS half of the system depends on. Access is nearly always the critical path, and I would rather find that out on day zero than in week two with a finished agent and no way to put it on a real line.

I also ask which system the humans look at all day, because that is where the agent's output has to land. An agent that books into a calendar the staff never open is worse than no agent.

Name the person who owns the words

Voice agents do not usually fail by hallucinating. They fail by going stale, quietly, because the price changed or a location moved and the agent kept confidently saying the old thing.

So I ask for a name. Who, on your side, is responsible for telling me when the facts change? If nobody wants to be that person, I design around it: fewer hard-coded facts, more live lookup, and a review trigger tied to events like a price change or a new location, rather than a calendar reminder nobody honours.

Agree what "working" means before building

The last section of the call is definitions, because "it works" is not measurable and every client means something different by it.

I want to know which calls count, since that denominator decides every percentage afterwards, and which single number they will judge this by in month one. Usually it is booked appointments from calls that would previously have been missed, or the share of calls resolved without a human. I write it down, in their words, with the exclusions attached.

Then I tell them what I will report and how, because the metrics that matter are not the ones the platform dashboard shows by default, and the agent's own self-assessment of a call needs auditing against real listening.

The red flags that make me shrink the scope or say no

A few answers reliably predict a bad project:

No single decision maker. Voice agents touch phones, calendars, brand voice and staff workflow. Four stakeholders with a veto and no owner produces an agent that is redesigned every week.
"Callers should not know it is not human." I do not build that. Beyond the compliance exposure, it fails the first time someone asks directly, and honesty in the greeting costs nothing in booking rate.
Everything is urgent and clinical. If most calls genuinely need human judgement, the honest recommendation is capture and routing, not resolution.
No willingness to test. A client who will not sit through a handful of test calls before launch discovers the problems through their own customers instead.
The real problem is staffing a process nobody has defined. If the humans do not agree on how a booking is taken, the agent cannot.

Saying that out loud on the first call has cost me a few projects and saved me several months.

What the first build should actually be

Almost every scoping conversation ends in the same recommendation: build the narrowest agent that plugs the biggest leak, put it live behind a forward rather than a port, and expand from evidence instead of imagination.

That usually means one intent, one calendar, one escalation path, and the complete automation layer behind the call done properly, because a call that goes perfectly and books nothing is the failure clients never forgive. It is far less than the brief asked for and goes live in a fraction of the time, which means version two is designed against real call recordings rather than a workshop.

The output is one page

The deliverable from all of this is a single page: the leak being plugged, the calls in scope, the calls explicitly out of scope, the never-do list, the systems and who owns access, the person who owns the words, the one number we are judging it by, and what version one deliberately does not include.

That page has done more for the quality of my builds than any prompt technique. The engineering behind a good voice agent is genuinely interesting, but almost none of the projects I have seen struggle were struggling because the engineering was hard.

I build production AI voice agents and the automation, CRM and telephony layer behind them for founders across the US, UK and Europe. More about my work here, or book a call.