TL;DR: The post-call summary is the only part of an AI voice agent most of the client's staff will ever see, and the default one is usually useless: a polite paragraph that retells the conversation in order. A summary people act on is written for the person who picks it up next, not for the record. My rules: lead with the outcome and the next action, generate it as structured fields instead of free prose, put the details a human has to act on in fixed slots, say plainly what the agent did not do, flag uncertainty instead of smoothing it over, deliver it where the staff already work, and keep the full transcript one click away for when someone needs the evidence. If a receptionist has to open the recording to find out what to do, the summary failed.
When a business switches on a voice agent, the owner listens to a few calls in the first week. After that, almost nobody listens to calls again. What the team actually reads, every day, is the note the agent leaves behind: in the CRM, in a dashboard, in a text or an email to whoever is on shift.
That note is the product, as far as most of the staff are concerned. I have built voice and chat agents for hospitals, real estate, vet clinics and car garages at Fortell AI, and for the CallSetter AI product at Tested Media, where the agents feed a client dashboard and GoHighLevel. In every one of those settings, the call itself could be excellent and the client could still feel the agent was not working, because the summary in front of them did not tell them what to do.
I have written about getting the details right during the call and keeping the post-call automation from silently failing. This piece sits between the two: what the human on the other side actually receives, and how to make it something they use.
Why the default summary does not work
Most platforms can produce a call summary out of the box, and most out-of-the-box summaries read like this: "The caller contacted the clinic regarding their dog. They mentioned the dog has been limping. The agent asked several questions and discussed availability. The caller indicated a preference for later in the week."
Every sentence is true and none of it is usable. The person reading it wants to know four things: who this was, what they need, what already happened, and what is left for a human. The narrative version hides all four inside a retelling of the call in the order it happened.
The cause is simple. A model asked to "summarize this call" optimizes for a faithful, readable account. A receptionist does not want an account. They want a task. Those are different documents, and you only get the second one if you design for it.
Rule 1: lead with the outcome and the next action
The first line of every summary should answer one question: what is the state of this call now? Booked, needs a callback, transferred, not a fit, spam, caller hung up. The second line should say what a human needs to do, or say explicitly that nothing is needed.
So instead of the paragraph above, the top reads:
- Outcome: Not booked. Caller wants Thursday or Friday after 4pm, nothing was available.
- Next action: Call back today to offer the first late slot or a waitlist place.
Everything else comes after that. Staff skim. Many will read the first two lines of a summary and nothing else, so those two lines have to carry the whole decision. If the answer to "do I need to do anything?" is buried in sentence four, a real share of callbacks simply never happen.
Rule 2: generate fields, not prose
The most important technical decision is to stop asking the model for a paragraph and start asking it to fill a fixed structure. In Retell this is post-call analysis with defined fields, and the same idea works on any platform: define the fields per client, give each one a type and a short instruction, and let the model populate them from the transcript.
A typical set for a service business looks like:
- outcome (one of a fixed list of values)
- next action and who owns it
- caller name, callback number, and the reason for the call in one line
- urgency (a fixed scale, with the client's own definition of urgent)
- whether a booking was made, and its reference if so
- anything the caller asked that the agent could not answer
Fixed values matter more than they look. An outcome field that can only be one of six values can be filtered, counted and routed. A free-text outcome cannot. Once the summary is structured, the dashboard can show "five callbacks needed today" at the top, the metrics worth tracking fall out of it naturally, and n8n can route urgent calls differently from routine ones without anyone parsing English.
You can still render a short human-readable note from those fields. But the fields are the source of truth, and the prose is a view of them.
Rule 3: put the actionable details in fixed slots
Phone numbers, names, dates, booking references and the specific thing the caller wants belong in their own fields, always in the same place, never inside a sentence. When a receptionist is calling people back between patients, they should not have to read a paragraph to find a number. The callback number sits in the same position on every note, and in a CRM it lands in the field the staff already dial from.
This also connects back to capture. A summary is only as good as the data under it, and if the agent misheard an email address during the call, a beautifully formatted summary just presents the wrong address with confidence. That is why the validation lives in the pipeline, before anything reaches the note.
Rule 4: say what the agent did not do
The most expensive misunderstanding I see is a summary that implies something happened when it did not. "Discussed appointment options" reads, to a busy person, like "booked". "Will pass the message to the vet" reads like the message has been passed, when in fact the summary is the message and nobody has seen it yet.
So the summary should state the unfinished parts in plain words: not booked, no payment taken, question about pricing not answered, caller expects a call back today. When an agent hands the call to a human, the note should say whether the transfer actually connected or fell through to voicemail, because those are completely different situations for the person reading it.
Calls where nothing actionable happened deserve the same honesty. A clear "wrong number, no action" or "spam, no action" lets staff clear the item in a second, and keeps the real tasks from drowning in noise.
Rule 5: flag uncertainty instead of smoothing it over
Models are good at producing confident text from messy input. That is exactly the wrong instinct in a summary. If the caller was hard to hear, if they gave two different dates, or if the agent was not sure it understood the reason for the call, the note should say so.
I give the analysis step explicit permission to be unsure: a field for "needs human review" with a short reason, and an instruction that a doubtful value should be marked as doubtful, not quietly picked. A receptionist who sees "caller said either the 12th or the 21st, confirm" makes one quick call and gets it right. A receptionist who sees a single clean date that happens to be wrong books the wrong day, and the business blames the agent, fairly.
Rule 6: deliver it where the staff already work
A perfect summary in a place nobody looks is the same as no summary. Before building anything, I ask where the team actually lives during the day. For CallSetter that meant the client dashboard and GoHighLevel, because that is where campaigns and leads were already managed, which I covered in the piece on the dashboard and CRM layer. For a small clinic or garage it might be a text message to the front desk phone, or an email to a shared inbox.
The routing should follow the outcome field. Urgent calls go somewhere that interrupts a person. Routine bookings go into the CRM quietly, because the booking already exists and nobody needs a notification for it. Callbacks go into whatever the team treats as a to-do list. One firehose channel that receives every call equally trains people to ignore it within a week.
Rule 7: keep the evidence one click away
Short summaries create a fair question: what if the summary is wrong? The answer is that the transcript and recording are always linked from the note, never pasted into it. The summary is for acting. The transcript is for checking. Pasting the full transcript into the CRM note turns a two-line task into a wall of text, and the two lines get lost again.
That link also makes the summary safer to trust. Staff who can verify a note in ten seconds when something feels off end up trusting the notes more, not less, because they have seen that the evidence is there and that it matches.
Testing summaries like any other output
Summaries regress just like the conversation does. A prompt change that makes the agent friendlier can quietly change how it phrases outcomes, and a field that used to say "Not booked" starts saying "Caller will consider options". So summary fields go into the same regression test set as everything else: a library of recorded or simulated calls, each with the outcome and next action I expect, checked whenever the prompt or the analysis instructions change.
The other test is the cheapest one. In the first weeks after launch, I sit with whoever reads the notes, or at least ask them, what they had to look up that the note should have told them. Every one of those answers becomes a field or an instruction. It is the most useful feedback loop in the whole first two weeks live, because it comes from the people who decide whether the agent is worth keeping.
The short version
- Write for the next person, not for the record.
- Outcome and next action first, in fixed values a dashboard can count.
- Details people act on go in fixed slots, validated before they get there.
- Say what did not happen as clearly as what did.
- Mark doubt as doubt.
- Route by outcome to where the team already works.
- Link the transcript, do not paste it.
The conversation is what impresses a client in the demo. The note on the screen the next morning is what they judge every day after that.
I build production voice and chat agents on Retell, wired into n8n, GoHighLevel and Twilio, including the summaries, dashboards and routing that the client's team actually uses. More about my background here, or book a call.