TL;DR: A live voice agent is software you cannot diff and cannot replay. You edit one sentence in the prompt to fix one complaint, and the change lands globally, in branches you were not thinking about, and nothing errors. Here is the change discipline I run on agents that are already answering real phones: a small frozen set of real call scenarios with written expected outcomes, config versioned outside the platform so a rollback is one action, changes shipped one at a time into the quietest traffic available, and a re-run triggered by the changes I did not make, like a model version bump, which are the ones that actually bite.
Launching a voice agent is the easy half. The uncomfortable half starts the week after, when the client sends a note saying the agent was a bit pushy on that last call, and you open the prompt to soften one line.
That edit is a production deployment. It just does not look like one. There is no build, no diff review, no test suite going red, and no obvious blast radius. The agent handles the next call using the new text, in every conversation branch, including the four you were not thinking about when you rewrote the sentence.
I have shipped enough of these changes to be nervous about them. This is the process I use instead of hoping.
The change you make is not the change you shipped
A prompt is global state written in prose. There is no scoping. A sentence added to soften the booking flow is also read by the model during the refusal, during the transfer, and during the part where a caller is spelling out a registration number.
Two things that have genuinely happened to me, in some form, on production agents:
A line added to make the agent more helpful about pricing made it more willing in general, and it started answering an adjacent question it was supposed to decline. Nothing in the change mentioned that topic.
A tightening of the confirmation step, meant to reduce mistakes on email addresses, made the agent slower and more repetitive on every other capture in the call, which showed up as callers hanging up mid-flow rather than as an error anywhere.
Neither of those was visible in the text I changed. Both were visible in behaviour, days later, in metrics nobody was watching that closely. And the failure was silent in exactly the way post-call automation failures are silent: the calls completed, the transcripts read fine, and no system reported a problem.
Why the usual safety nets are missing
If this were a normal service, you would lean on tests, a diff and a deterministic re-run. On a voice agent you have none of the three.
There is no deterministic replay. You cannot re-run last Tuesday's call against the new prompt. The audio, the timing, the interruptions and the model sampling are all gone. This is the single hardest adjustment coming from other real-time work, and the one I called out when writing about moving from Unity to AI engineering: games hand you a repro, calls hand you a story.
The output is a conversation, not a value. Assert on exact wording and the test is useless, because the wording is supposed to vary.
Production config is edited live in a browser. Most voice platforms let anyone with a login rewrite the agent in a text box and publish immediately. There is often no review step, and the change history is whatever the vendor decided to keep.
Part of the stack updates without you. The model version, the voice, the recognition behaviour. You did not ship anything and the agent changed anyway.
So the discipline has to come from the outside.
Build a small regression set out of real calls
The most useful artifact I keep for any live agent is a frozen list of twelve to twenty call scenarios, taken from real traffic rather than imagined.
Where the scenarios come from
I pull them from the actual call distribution, the same way I pull the knowledge tiering during scoping. The set needs to cover:
- the two or three highest-volume intents, in their plain form
- every refusal the agent is supposed to make, one scenario each
- one messy capture: an awkward email, a surname that gets heard three ways, a postcode
- one caller who interrupts and one who changes their mind mid-booking
- one call that must end in a transfer, and one where the transfer destination does not answer
- one out-of-scope question that should end in an honest "I do not know, let me get someone"
That is not exhaustive coverage. It is the set where a regression is most likely to be both plausible and expensive.
Write down what each scenario must do
The trick is asserting on observable outcomes rather than phrasing. For each scenario I write, in one or two lines:
- which exit the call must reach (booked, message taken, transferred, refused)
- which tools must be called, and with what arguments
- what must exist afterwards: the calendar event, the CRM record, the specific fields captured correctly
- what must never appear: a price quoted, a diagnosis offered, a claim of being human
That is a scoring rubric per field, which is the same standard I apply to the metrics that matter. Pass or fail per assertion, not an overall impression of whether the call felt good.
Automate what you can, listen to what you cannot
I split the run in two, because the two halves catch completely different regressions.
The automated half is the pipeline I described in building and testing agents with Claude Code and Comet: drive the agent through the scenarios, capture the transcripts and the resulting records, and check the assertions. This reliably catches the structural regressions, a tool that stopped being called, a field that stopped being captured, a refusal that stopped happening, an exit that now lands somewhere else.
The human half is a small number of calls listened to on a real phone line. This catches everything a transcript cannot show you, which is most of what clients complain about: pace, a business name suddenly mispronounced after a voice update, a confirmation read as a wall of digits, an agent that now sounds curt. The transcript contains the correct string in all of those cases, which is exactly why I keep insisting on an editing pass done by ear.
Two tiers keeps this affordable. A wording fix inside one knowledge base answer gets a three-scenario smoke run. Anything touching the prompt, the flow structure, the tools or the model gets the full set.
Make the change reversible before you make it
Before a change goes anywhere near live traffic, I want three things true.
The config lives in version control, not only in the platform. I export the prompt, the flow, the tool definitions and the automation workflows into the repo and commit them with the reason for the change in the message. The platform's own history is a convenience. The repo is the audit trail, and it is what tells you six weeks later why that odd sentence is in there.
One person publishes. Editing is fine, publishing is not a group activity. Nearly every unexplained "the agent got worse this week" I have investigated turned out to be two people making reasonable edits to the same prompt two days apart.
**Rollback is one action, and it has been rehearsed once.**Restoring the previous version should take under a minute while the phone is ringing. That habit came out of shipping mobile releases at RAQTS through App Store and Play Console gatekeepers, where the only real protection against a bad build is how quickly you can pull it.
And ship one change at a time. Batching six edits into a single publish means that when the transfer rate doubles, you have six suspects and no way to attribute.
Ship into the quietest traffic you have
There is no staging environment with real callers, so the closest thing is a smaller slice of real traffic.
In practice that means publishing at the start of a quiet window rather than at nine on a Monday, or putting the change on one agent when a client runs several, or exposing it first on an overflow line that only takes the calls reception could not reach. Then I read the first fifteen or twenty live calls myself. Not a random sample later, the first ones, immediately.
After that, the monitoring signal is a change in shape rather than an error. Transfer rate up, capture accuracy on one field down, average call length up by twenty seconds, resolution rate down while call volume is flat. Those are the fingerprints of a regression, and none of them throw an exception.
The change you did not make
This is the part clients find genuinely surprising, and it is the reason a regression set is worth keeping rather than running once.
The model version changes. The voice provider updates a voice and the business name is suddenly pronounced differently. Recognition behaviour shifts slightly. A client's staff member edits the knowledge base, correctly, and creates a contradiction with a line in the prompt. A calendar or CRM API changes a field. Any of these can alter agent behaviour with no deployment on your side at all.
So: pin versions wherever the platform lets you, treat any provider change notice as a trigger to re-run the full set, and keep a dated note of when the set last passed. When a client says the agent has got worse and you can say it passed everything eleven days ago and here is what changed since, the conversation is an investigation instead of an argument.
That is also why staleness, rather than hallucination, is the failure mode I plan around in the knowledge layer. A stale fact and a silent regression have the same signature: everything sounds right.
The short version
Keep the set small and real. Assert on outcomes, not wording. Automate the structural checks and listen to the rest on an actual phone. Version the config outside the platform, publish through one person, and rehearse the rollback. One change at a time, into quiet traffic, with the first calls read by you. Re-run when the provider changes something you did not.
None of it is sophisticated. It is the same release discipline any production system gets, applied to a system whose test failures arrive as a polite customer who did not get a call back.
I build production AI voice agents and the automation, CRM and telephony layer behind them for founders across the US, UK and Europe. More about my work here, or book a call.