TL;DR: Most voice agents that fail in production do not fail at conversation. They fail at facts. The agent says the price confidently, and the price changed in March. The fix is not a bigger prompt, it is deciding where each fact lives before you write a word of script. I use three tiers: a small set of stable facts baked into the prompt, a knowledge base for the long tail of things people occasionally ask, and a live lookup for anything that can change today. The rule I apply to every fact is one question: if the agent is wrong about this, who pays for it?
A client sends me their agent because "it keeps making things up." I listen to five calls and it has not made anything up at all. It repeated, accurately, what someone pasted into the prompt eight months ago. The business changed and the agent did not, because nobody decided who owned that sentence.
That is the knowledge problem, and it is separate from every other part of a voice build. I have written about how to structure the conversation, how to keep it fast, and when to hand off. None of that helps if the agent is fluent and wrong. Here is how I design the layer underneath.
The failure is staleness, not hallucination
Hallucination gets the attention because it is dramatic. In practice it is the smaller problem. A modern model with a clear instruction not to invent facts is reasonably well behaved, and a caller can usually tell when an answer sounds made up.
Staleness is worse precisely because it sounds correct. The agent gives a firm, well-phrased answer with the right tone and the right level of detail. The caller believes it, arrives at the clinic on a day it is closed, or shows up expecting a price that no longer exists. Nobody on the call detects a problem. The damage surfaces later, at the counter, in front of a human who has to explain that the phone system was wrong.
So the question I ask about every fact in an agent is not "will the model get this right." It is "what happens the day this becomes untrue, and who notices." That question sorts facts into three places on its own.
Tier one: what belongs in the prompt
Put a fact in the prompt only if it is stable, short, and asked constantly.
For most businesses that is a list of about ten things: what the business does, opening hours, address and rough parking or access notes, whether they take walk-ins, the main services by name, and the two or three things people always ask that never change. Identity and tone go here too, along with every boundary the agent must not cross.
Prompt facts are fast, because they cost nothing at call time, and that speed is why the high-frequency answers belong here. They are also the most dangerous kind of fact, because they are invisible. Nobody on the client's team is ever going to read the prompt again. Keeping this list short is not a style preference, it is risk control: ten facts you can audit in a minute, not a page of copy-pasted website text nobody will revisit.
One habit that has saved me repeatedly is putting every prompt fact in a single clearly marked block with a date on it, so when a client asks why the agent said something, the answer takes ten seconds to find rather than a scroll through a wall of instructions.
Tier two: the knowledge base for the long tail
The next tier is everything a caller might ask that is real, answerable, and not worth a prompt slot: do you handle this specific breed, is that brand of part covered, what is your policy on cancellations, do you validate parking, what happens if I am late.
Two things about knowledge bases for voice specifically, both of which people get wrong because they are used to chat.
Write the chunks as spoken answers, not as documents. A knowledge base assembled by dumping a PDF or scraping the website gives the model paragraphs written to be read. Retrieval then hands the model three overlapping page fragments and it produces a forty second monologue with subheadings audible in it. What works is one question, one answer, two or three sentences, written the way a good receptionist would say it out loud. Retrieval quality goes up as a side effect, because each chunk is about exactly one thing.
Prefer fewer, cleaner entries over completeness. Every entry is a fact somebody has to maintain. A knowledge base with forty accurate entries beats one with four hundred entries of which fifty are quietly wrong, because you cannot tell which fifty. If a question comes up twice a year and the answer changes, it is better handled by an honest "let me get someone who can confirm that for you."
Tier three: anything that can change today
The third tier is not knowledge at all. It is a live lookup against whatever system already holds the truth, called mid-conversation.
Availability is the obvious one, and I covered the mechanics of that in booking against a calendar. But the same rule catches a lot more: order or repair status, whether a specific item is in stock, whether a person is already a customer, current pricing where pricing is not fixed, and anything at all attached to a particular caller's record.
The test is simple. If a fact can differ between the morning and the afternoon, the agent must not carry it. It has to ask something that knows.
This tier is expensive in the only currency a phone call has, which is the caller's patience, because a lookup happens while they sit in silence. That cost is exactly why the tiering matters. You spend a live lookup on the handful of questions that genuinely require one and answer everything else instantly.
The maintenance problem is the real problem
Here is the part that decides whether an agent is still trustworthy in month six. A knowledge layer is not a build artifact. It is a business document that decays.
What I set up on every project:
- A named owner on the client's side. Not the agency, not me. The person who would change the sign on the door is the person who owns the agent's facts. If nobody will own it, that is worth knowing before launch rather than after the first wrong answer.
- A single editable surface. The client updates facts in one place they can actually reach, and it propagates. If updating the agent requires messaging the builder, the agent will be out of date within a quarter, guaranteed.
- A review trigger tied to real events, not the calendar. Price change, new service, staff change, holiday hours, a policy change. A quarterly review that nobody remembers is theatre. A rule that says "when the price list changes, the agent gets updated in the same hour" is a process.
- A log of unanswered questions. Every call where the agent had to punt is a candidate knowledge base entry. This is the single highest value input into the knowledge layer and almost nobody collects it. I write these out of the post-call analysis into the same n8n workflows that handle the rest of the after-call work, so the gap list builds itself.
That last one turns the knowledge base from a thing you guess at up front into something the callers themselves specify over the first month.
Refusing well is part of the knowledge layer
An agent that answers everything is not a well-informed agent, it is an unbounded one. Grounding has to be explicit: answer from what you were given, and when you do not have it, say so and route.
The phrasing matters more than people expect, because "I do not have that information" sounds broken while "I would rather not guess at that, let me get you to someone who can confirm" sounds careful. Same behaviour, completely different impression, and the second one is what a good human receptionist actually says.
This is the same competence-boundary discipline I use for escalation, and it matters most in the verticals where a confident wrong answer is genuinely harmful. In the medical clinic agents I built at Fortell AI, the knowledge layer's most important property was not coverage. It was refusing cleanly and instantly on anything clinical.
I also picked this up somewhere unexpected. Building clinical eye-tracking diagnostics at Nystag, the rule was that a trace admits where it has no data instead of drawing a plausible line through the gap. A voice agent that says "I do not know" is the same instrument design.
How to test it
Conversation testing and knowledge testing are different exercises, and the second one gets skipped.
Three passes I run, using the Claude Code and Comet pipeline to batch them rather than dialing by hand:
- Questions with known answers. Boring, necessary, catches retrieval that pulls the wrong chunk.
- Questions the knowledge base does not cover. More important than the first pass. This is where you find out whether the agent refuses or improvises, and improvising here is the defect you actually shipped.
- Questions phrased the way real callers phrase them. Nobody asks "what is your cancellation policy." They ask "what happens if my car is still not ready and I need to move it." Retrieval that only works on the tidy phrasing works on nobody.
Then, after launch, the honest check: pull the calls where the agent gave a factual answer and read a sample against the current truth. Not the transcripts the agent scored as successful, a random sample. That is the same auditing habit I apply to outcome metrics, for the same reason. A system reporting on itself grades generously.
The short version
- Sort every fact by what happens when it becomes untrue, before writing any script.
- Prompt: stable, short, constantly asked. Keep the list small enough to audit.
- Knowledge base: the long tail, written as spoken answers, one question per chunk, fewer and cleaner.
- Live lookup: anything that can change today or is specific to this caller.
- Name an owner on the client side and give them one place to edit. No owner means decay.
- Log every unanswered question and let callers write your backlog.
- Make refusal a designed behaviour with good phrasing, not a failure state.
- Test the questions you did not answer, not just the ones you did.
Conversation quality is what people notice in a demo. Factual reliability is what decides whether the agent is still on the phone line a year later.
I build production voice and chat agents on Retell, wired into n8n, GoHighLevel and Twilio, with the knowledge, CRM and reporting layer behind them actually working. More about my background here, or book a call.