How is writing a prompt for a voice agent different from writing a chatbot prompt?

The medium changes almost every rule. A chat user can re-read the previous message, skim a list of eight options and scroll back to a detail they missed. A caller can do none of that, so anything longer than a couple of sentences starts dissolving before it lands, and a spoken menu of more than three items is a memory test rather than a choice. Phone audio is also narrowband, which strips out exactly the clarity that makes a long sentence survivable in person. In practice that means one idea per sentence, one question per turn, lists converted into a choice of two, and the important clause placed first so an interruption does not destroy the turn.

Should you script an AI voice agent word for word or describe what it should do?

Both, split by consequence. I quote exact wording for anything where the specific words carry risk or set the tone: the greeting, the call recording disclosure, any compliance or regulatory line, the refusal phrasing for questions the agent must not answer, and the handoff to a human. Everything in the middle is written as behaviour, meaning what the agent is trying to find out and what counts as done, rather than as dialogue. Full scripting sounds robotic and shatters the moment a caller goes off path, while pure behavioural description sounds natural right up until the agent improvises something it had no business saying.

Why does an AI voice agent mispronounce times, prices and business names?

Because text to speech reads what the model hands it, and the model writes for a page. Unless the prompt says otherwise you get times in a format that does not match the region, currency symbols read as symbols, long reference numbers spoken as one unbroken run of digits, and URLs or email addresses read aloud when they should have been sent as a text message. Business and clinic names are the most common failure and the first thing a caller hears, so the fix is to write the phonetic spelling directly into the prompt. None of this shows up in a transcript, because the transcript contains the correct string. You only catch it by listening to the agent in the real voice over a real phone line.

Write Your Voice Agent Prompt for the Ear, Not the Page

TL;DR: Most bad voice agents are not badly engineered, they are badly written. The prompt was written to be read, and it is being heard. A caller has no scrollback, no bullet points and no patience, so the rules change: one idea per sentence, one question per turn, never more than three options out loud, spell out how numbers and times should be spoken, script the lines that carry risk and describe the rest as behaviour, and keep the agent's own turns short so a human can interrupt them. Then test by ear, in the real voice, over a real phone line. A prompt that reads beautifully in a text box can sound unbearable at eight kilohertz.

The first production agent I ever had rejected by a client failed for a reason that had nothing to do with the model. The logic was right, the booking worked, the transcript read like a competent receptionist. On the phone it sounded like someone reading a terms and conditions page at speed. The client's exact words were that it did not sound like a person, it sounded like a website.

That is a writing problem, and it is the single most common one I see. Everything below is what I now do differently across the Retell agents I have built at Tested Media and Fortell AI for hospitals, vet clinics, brokerages and garages.

The page and the ear are different mediums

Language models write like documents, because they were trained on documents. Left alone, a prompt produces headings, parenthetical asides, hedged multi-clause sentences and lists of six items. All of that is fine on a screen. Over a phone line it fails for three specific reasons.

There is no scrollback. A reader who loses the thread glances up a line. A caller who loses the thread has lost it permanently, and their only recovery move is to interrupt and ask you to repeat, which costs the call ten seconds and costs you the caller's confidence.

Working memory is small and the clock is running. People hold about three spoken options before the earlier ones start dissolving. Read out eight appointment types and the caller remembers the last one and picks it, which is not a choice, it is fatigue.

And the audio pipe is bad. Telephony is narrowband, so the crispness that makes a long sentence survivable in person is simply not transmitted. Everything you write is being heard through the worst audio equipment your caller owns.

Rule one: one idea per sentence, one question per turn

The fastest improvement available in any voice prompt is cutting the sentences in half and asking one thing at a time.

A written line like "Can I take your name and a contact number, and let me know whether you would prefer morning or afternoon?" contains three requests. The caller answers one and a half of them, and now the agent has to run a repair turn to collect the rest, which is slower than asking twice would have been.

I write the script lines as if they will be spoken by someone who is busy but polite. Short declarative sentence, then a question, then stop. Stopping matters more than people expect: a clean question at the end of the turn is also what makes endpointing work, because both the model and the human on the other end use the same cue to know whose turn it is.

Rule two: never read a list out loud

Anything that would be a bulleted list on a page has to become a choice of two, occasionally three, out loud.

The clinic that offers eleven service types does not get an eleven item menu. It gets a question that narrows: is this for a check up or for something that is bothering you today. The branch that follows can offer two more. A caller who wants something outside the tree will say so unprompted, and the agent needs a path for that, but designing for the exception at the top of the call punishes the ninety percent who wanted the common thing.

The same applies to availability. When the agent has fetched a calendar, it should offer the best two slots and hold the rest in reserve, not recite everything the API returned. I go into why that also protects the booking flow elsewhere, but the reason here is purely acoustic: a list is unlistenable.

Rule three: write how it should be said, not how it is written

Text to speech reads what you hand it, and what you hand it is often not what you want heard. This is where I spend a surprising amount of prompt space.

Times. Decide whether the agent says two thirty in the afternoon or fourteen thirty, and write the rule explicitly, matched to the region. UK clients and US clients want different answers here.
Dates. Say the weekday alongside the date. Thursday the fourteenth is far harder to get wrong than the fourteenth, and it doubles as a checksum the caller can catch.
Numbers and money. Say how prices and reference numbers should be grouped and whether currency is spoken before or after. Long digit strings need explicit grouping instructions or they come out as one unbroken run.
Symbols and abbreviations. Never let the agent read a URL, an email address or an ampersand aloud unless you have told it exactly how. Anything with a slash in it belongs in a text message, not in speech.
Names of the business. Clinic and brand names are the most common mispronunciation, and they are the first thing a caller hears. If the voice gets it wrong, write the phonetic spelling into the prompt and keep testing until it is right.

None of this appears in a transcript. A transcript shows the correct string. Only the audio tells you the agent just said the pound sign out loud.

Rule four: script the risky lines, describe everything else

There is a real tension between over-scripting, which sounds robotic and shatters the moment a caller goes off path, and under-scripting, which sounds natural right up until the agent improvises something it had no business saying.

My split is by consequence. Exact wording, quoted verbatim in the prompt, for anything where the specific words carry risk or set the tone: the greeting, the recording disclosure, any compliance or regulatory line, the refusal phrasing for questions the agent must not answer, and the handoff to a human. Behavioural description, not a script, for everything in the middle: what the agent is trying to find out, what counts as done, when to move on.

That split also keeps the prompt short, which matters for a reason beyond style. Every extra paragraph of instruction is tokens on the critical path of the response and one more competing directive for the model to weigh, so bloated prompts are both slower and less obedient. I keep per node prompts tight for exactly that reason in conversation flows, and the same discipline pays off in latency.

Rule five: write turns that survive being interrupted

Real callers talk over the agent. That is not a defect, it is how phone conversations work, and it is why a long agent turn is a design mistake rather than just a bore.

Keep the sentence carrying the important information first and the pleasantries after, so if the caller barges in halfway they have already received the part that mattered. Never bury the question at the end of a five sentence paragraph. And when the agent has to wait on something, a calendar lookup or a CRM write, have it say a short line to cover the pause rather than going silent, because silence on a phone call reads as a dropped connection within about two seconds.

Personality instructions deserve a warning here. Writing "be warm and friendly" reliably produces padding, because to a language model warmth looks like extra words. Specify behaviour instead: acknowledge what the caller said before asking the next question, use the caller's name once, do not apologise more than once per call. Concrete behaviours produce warmth. Adjectives produce filler.

Rule six: edit with your ears

The final pass is not reading the prompt, it is listening to the agent.

I test in the actual selected voice, over a real phone number rather than the web widget, because the browser tester hides both telephony compression and the real endpointing behaviour. I run the messy cases through the Claude Code and Comet pipeline rather than dialing everything by hand, but the first listen is always me on a phone, and I listen for four things.

Where did I stop paying attention. Where did I want to interrupt. Where did the agent say something no human receptionist would ever say out loud. And where did it sound like it was reading.

Then cut. Nearly every edit I make on a second pass is a deletion. If you are unsure whether a sentence earns its place, it does not, because on a phone call every sentence is charged to the caller's patience.

The short version

Write for a listener with no scrollback: one idea per sentence, one question per turn, then stop.
Turn every list into a choice of two, and offer the best two calendar slots rather than all of them.
Spell out how times, dates, numbers, symbols and the business name should be pronounced.
Quote exact wording only for the greeting, disclosures, refusals and handoffs, and describe behaviour everywhere else.
Keep prompts short, because length costs latency and dilutes instruction following.
Front load the important clause so an interruption does not destroy the turn, and cover waits with speech.
Replace personality adjectives with concrete behaviours.
Edit by ear on a real phone line, in the real voice, and delete more than you add.

The model is not what makes an agent sound human. The writing is. A mediocre model with a well written script beats a frontier model reading a document, every single time.

I build production voice and chat agents on Retell, wired into n8n, GoHighLevel and Twilio, including the script, the pipeline and the reporting layer behind them. More about my background here, or book a call.