TL;DR: A tool your voice agent calls in the middle of a phone call is not an API call, it is a turn in a conversation with a human waiting through it. That changes the design. Every in-call tool needs a deadline shorter than the caller's patience and a line to say when it expires, a return value shaped to be spoken rather than a record to be parsed, parameters filled from the call context instead of from what the caller said, an empty result treated as an answer rather than an error, and a hard split between the tools that read and the tools that change something. Everything that does not have to happen before the agent speaks again belongs after the call, not during it.
Most of the writing about voice agents stops at the prompt. In production the prompt is the part that rarely breaks. The part that breaks is the layer where the agent reaches out to something real: checking whether Thursday at four is free, looking up whether the car is ready, finding out if this caller is already a customer, putting a booking into a calendar. On the builds I do that layer is a set of custom functions on the Retell side pointing at n8n webhooks, with GoHighLevel or a practice management system on the far end of them. Same shape at Tested Media on CallSetter AI, same shape on the UK vet, garage and estate agent lines I built at Fortell AI.
I have spent more debugging time in that layer than anywhere else, and almost none of it was the model choosing the wrong function. It was tools that behaved perfectly as APIs and terribly as conversation.
A tool call is a turn, and the caller is listening to it
When a backend service calls an API, nothing is waiting except a process. When a voice agent calls a tool, a human being is holding a phone to their ear in silence. Six hundred milliseconds is fine. Three seconds is a pause people fill by saying "hello?", which then arrives as a new caller turn in the middle of your function call and drags the conversation sideways.
So the first design question for any tool is not what it returns. It is what the caller hears while it runs. That means the tool count per node stays small, the wait gets covered with speech before the call fires, and anything that does not change what the agent says next never runs during the call at all.
That last one is the rule people break most. If the tool result does not change the agent's next sentence, it is not an in-call tool. It is a post-call automation that you have moved onto the critical path for no reason.
Give every tool a deadline and a line to say when it expires
Every in-call tool gets an explicit timeout, set well below where a caller starts talking over the silence. I treat two to three seconds as the working ceiling for a lookup, and I would rather return a degraded answer inside that window than a perfect one outside it.
The important half is what happens at the deadline. A timeout is not an exception in a voice agent, it is a return value, and it needs a conversational answer waiting for it. "I am having trouble reaching the diary right now, can I take your number and have someone confirm within the hour?" is a good outcome. Dead air followed by the model improvising is not, and an improvising model in a booking flow will confidently invent an availability slot.
So each tool has three defined outcomes rather than two: it worked, it found nothing, or it did not answer in time. All three need a scripted line. If you have not written the third one, you have not finished the tool.
Return something speakable, not a record
The default instinct is to return the API response and let the model figure it out. It costs latency, it costs tokens on every subsequent turn because that blob stays in context, and it puts a pile of data in front of a model whose job is to talk.
I shape the return at the n8n end instead. An availability lookup does not return the calendar, it returns at most three options, already rounded, already in the caller's frame of reference, with the day name spelled out so the agent does not have to convert a timestamp out loud. A customer lookup does not return the contact record, it returns the two or three fields the next sentence needs.
Two things get better at once. The agent speaks correctly more often, because saying times and numbers out loud is where scripted phrasing beats improvisation. And the agent cannot leak what it was never given, which matters more than it sounds: a tool that returns the whole record has handed the model a decision about what to read out, and that decision belongs in your code, not in a prompt asking it nicely to be discreet.
The call already knows things, so stop asking the model for them
The parameters a tool accepts are a security boundary and an accuracy boundary at the same time.
A lookup tool should not take a phone number as a parameter. It should use the number the call arrived on, injected by the pipeline. A multi location business should not have the agent decide which branch it is talking about; the number dialled is known before the greeting, and the location identifier travels with the call as a variable, which is the same reason most businesses need one agent rather than six.
The general rule: if the pipeline knows a value, the pipeline supplies it. The model only fills parameters that genuinely came from the conversation, and every one of those has been through speech recognition, so it needs validating in the automation rather than in the prompt. A registration plate or an email address collected by ear is a candidate, not a fact.
Empty is an answer
The single most common in-call tool failure I see is a lookup that finds nothing and returns an error, or worse, returns an empty array with no instruction attached. The agent then either apologises for a system problem that did not happen, or stalls.
No availability on Thursday is a completely successful call to the tool. It should return the fact plus the next best action, so the agent has somewhere to go: the two nearest alternatives, or an offer to take details for the waiting list. Same for a customer lookup that finds no match, which is not a failure but a signal that this is a new caller and the flow should branch.
Write the empty case into the tool contract deliberately and the conversation stays on rails. Leave it undefined and you get the specific failure mode that annoys callers most, which is an agent that sounds broken when the answer was simply no.
Split the tools that read from the tools that change something
Read tools and write tools deserve different rules, and merging them is how double bookings happen.
Read tools should be cheap, repeatable and safe to call twice. Write tools should be narrow, idempotent, and carry the call identifier so a retry cannot create a second appointment. A booking write should check for a conflict immediately before it writes rather than trusting an availability check from ninety seconds earlier, because the double booking is a race rather than a prompt failure.
I also keep cancel and reschedule as separate tools from booking, matched against the record for the caller rather than against any reference the caller reads out, for the same reason you would not let a web form delete a record based on an unverified identifier.
And a transfer is a tool too. It should accept a destination from a fixed configured list, never a number assembled during the conversation, and it should carry the structured handoff payload so the human who picks up is not starting from nothing.
Errors need a script, not a stack trace
When a tool fails, whatever it returns is going into the model's context, and the model will try to use it. Return a raw error string and you eventually get an agent reading an HTTP status code to a caller. I have heard it happen.
So the failure return is a short, plain, speakable message plus a flag the flow can branch on. The agent's job is to stay useful in a small number of ways: take the details and promise a callback, offer the alternative that does not need the broken system, or transfer. Which one it picks should be decided by you at build time, per tool, not chosen live.
The other half is that the caller getting a graceful apology is not the end of the incident. A failed tool call needs to become a visible task for a human rather than a log line nobody reads, because a lapsed credential or a deactivated workflow will fail politely on every call until someone notices.
How I test the tool layer
Separately from the conversation, first. Each tool gets exercised directly with a good input, an empty result, a malformed input, an injected delay past the timeout, a revoked credential, and the same write sent twice. That is a fast loop with no phone call in it and it catches most of what would otherwise show up as a strange conversation.
Only then do I test through the agent, with the tools stubbed to be slow or to fail on demand, because what I am checking at that point is the talking rather than the plumbing. Both sets go into the frozen scenario set that gets re-run whenever the platform, the model or the prompt changes, since a model update can absolutely change how eagerly an agent reaches for a tool.
The short version
Keep the agent thin and the tools boring. Small number of tools per node, each one earning its place by changing what the agent says next. A deadline and a scripted line for every one of them. Returns shaped for the mouth rather than for a parser. Parameters supplied by the call wherever the call already knows. Empty treated as an answer. Reads and writes kept apart, with writes idempotent and conflict checked at the moment of writing. Everything else pushed behind the call into automation where it can retry, fail loudly and be fixed without a caller listening.
The conversation is the part everyone demos. The tool layer is the part that decides whether the conversation was worth having.
I build production AI voice agents and the tool, automation and CRM layer behind them for founders across the US, UK and Europe. More about my work here, or book a call.