TL;DR: A business phone line carries a lot of calls that are not customers: sales pitches, suppliers, recruiters, delivery drivers, wrong numbers, robocalls and silent connects. An AI receptionist that treats every one of them as a lead wastes minutes, pollutes the CRM and books fake appointments. The fix is a small, explicit "not a customer" branch: identify purpose early, give each type one short, polite exit, route suppliers and partners to a message, never transfer a cold pitch to a human, and keep these calls out of the pipeline and the metrics. I build this branch into every inbound agent now, across the hospital, vet clinic, car garage and real estate agents I built at Fortell AI and the CallSetter AI agents at Tested Media.
Read a week of transcripts on a newly live agent and the surprise is rarely the hard customer calls. It is how many calls are not customers at all, the cast I listed in the first two weeks live. A human receptionist handles them on reflex in ten seconds. An agent designed only for customers handles them badly, and the damage spreads well past the call.
Why non-customer calls are an engineering problem, not a nuisance
A sales call that lasts three minutes is annoying. The real costs show up downstream:
- Minutes cost money. Voice platforms bill per minute. An agent that politely engages a cold caller pitching SEO for four minutes is paying to be sold to. Across a month, the junk can be a noticeable slice of the bill I break down in what an AI receptionist really costs.
- The CRM fills with fake leads. If every call with a name and number becomes a contact and an opportunity, the pipeline the client's staff work from is now full of salespeople. The GoHighLevel structure I build around agents only stays useful if what enters it is real.
- Calendars get fake bookings. A salesperson who says "sure, put me down for Thursday" takes a slot a real customer then cannot.
- Staff get transferred junk. An agent with a generous transfer rule will happily put a cold pitch straight through to the owner's mobile, which is precisely the call the client bought an AI receptionist to avoid.
- Metrics lie. Booking rate, average call length and escalation rate all get distorted when a fifth of the calls were never customers. I made the case for separating them in the voice agent metrics that matter.
This is not about being rude to callers. It is about the agent knowing, early, which job it is doing on this call.
The cast of calls that are not customers
Not all non-customer calls want the same treatment. The ones I plan for on every inbound build:
Sales pitches and cold outreach
Marketing agencies, software vendors, merchant services, energy brokers. They usually ask for the owner by name or role, and they are good at sounding like they have an existing relationship.
Suppliers, partners and other businesses the client really works with
The parts supplier calling a car garage, the lab or referral partner calling a vet clinic, another agent's office calling about a property viewing. These are not customers, but they matter, and some are time-sensitive. Treating them like spam is a real mistake.
Recruiters and job seekers
Low urgency, but a business may want job enquiries captured somewhere.
Delivery drivers and couriers
"I'm outside, is anyone there to sign?" Real-time and physical. It needs a quick hand-off or a clear instruction, not a message for tomorrow.
Wrong numbers
One sentence and a goodbye.
Robocalls, silence and automated junk
Prerecorded messages, silent connects and occasionally another bot. These need a fast, cheap end and nothing else.
The design: a purpose check, then one short exit per type
The pattern I use is simple and it lives early in the agent's flow. On Retell that is usually a dedicated set of nodes in a conversation flow, or a clearly separated section of a single-prompt agent.
1. Find out why they are calling before collecting anything
Customer-only agents tend to open by asking for a name and number. Once the agent has asked for a name, it is in data-capture mode and keeps going. The opening should establish purpose first, "How can I help today?", and let the answer decide the branch. Details get captured only once the agent knows what kind of call this is, the same principle behind capturing caller details only when they serve the outcome.
2. Give each type exactly one exit
For each non-customer type, I write a single, short, specific response, not a general instruction to "handle sales calls appropriately". Specific beats general on a phone agent every time, which I covered in writing prompts for speech. For example:
- Sales pitch: "Thanks for calling. We don't take sales calls by phone, but you're welcome to email the details to the address on our website. Have a good day." Then end the call.
- Supplier or partner: take a short message with company, name, callback number and what it is about, tag it as a supplier message, and confirm it will be passed on.
- Recruiter or job seeker: point them to the website or a careers email, or capture a one-line message if the client wants that.
- Delivery driver: follow the client's instruction, which is usually a transfer to the front desk or a clear "please leave it at the side door" style answer.
- Wrong number: "I think you may have the wrong number, this is [business name]. Have a good day."
The exit should be polite and final. An agent that apologises, explains and asks "is there anything else?" is spending minutes on a call that was over in the first sentence.
3. Never transfer a cold pitch to a human
This is the rule clients thank me for most. Transfers are expensive, both in money and in a staff member's attention, and the human handoff design should only fire for callers who genuinely need a person. A salesperson asking for "the owner" by name is not an emergency, and "it's a personal matter" from someone who will not say what it is about is a well-known sales tactic. The agent offers a message instead. If the call is real, a message gets it to the right person anyway.
4. Let the agent hold the line politely
Persistent callers will push: "I just need two minutes with the manager", "it's about your account". The agent repeats its one exit calmly once or twice, then ends the call. It does not negotiate or get talked into a booking. Social-engineering pressure is a mild cousin of the attacks I covered in voice agent security and prompt injection, and the defence is the same: the agent has a fixed set of actions and talking harder does not unlock new ones.
5. End the junk fast
For silence and robocalls, the right behaviour is a short timeout: if nobody speaks after the greeting and one prompt, or the caller is clearly a recording, end the call.
Keep them out of the pipeline, not out of the record
The part teams forget is what happens after the call. Every call should still be logged, because "the agent hung up on someone" needs to be checkable. But non-customer calls should be classified and kept out of the places real customers live.
In the post-call webhook, alongside the usual outcome field, I add a call category with a fixed set of values: customer, supplier, sales, recruiter, delivery, wrong number, junk. The automation then routes on that field:
- Customers go through the normal pipeline: contact, opportunity, booking, follow-up.
- Suppliers and partners go to a separate notification, often a short message to a specific staff member, without creating a sales opportunity.
- Sales, recruiters, wrong numbers and junk are logged with their category and nothing else. No contact created, no pipeline stage, no follow-up SMS.
That last point matters. A "thanks for calling, book here" text to a cold caller is wasted, and texting numbers that never asked for it hurts the sender's reputation with carriers, as I covered in the A2P 10DLC guide.
The category also goes on the call summary, so the client's dashboard shows "4 supplier messages, 11 sales calls declined" as its own line instead of mixing them into lead counts. Clear, structured summaries are the whole point of call summaries staff actually read.
Do not over-filter
The opposite failure is real too. An agent tuned too aggressively starts declining genuine callers who sound like salespeople: a business calling a garage about a fleet of vans, a pet insurer calling a vet clinic about a real claim.
A few guardrails I use:
- Default to "message" when unsure, not "decline". If the agent cannot tell whether a caller is a supplier, a partner or a pitch, it takes a short message. A wasted message costs seconds. A lost customer costs a lot more.
- Give the agent the client's real partner list. A short list of known suppliers, labs, insurers or referral partners by name, as part of the knowledge base, lets the agent recognise legitimate business calls instead of guessing.
- Sales to the business is not the same as sales by the business. A caller asking to buy something is a customer, even if they use the word "quote" or "pricing". The decline branch is for people trying to sell to the client, not people trying to buy from them.
How to tune it after go-live
The non-customer branch is one of the first things I check in a live agent's first week:
- Pull every call classified as sales, wrong number or junk and skim them. Any real customer in that pile is a bug, and it gets fixed first.
- Pull every short customer call under twenty seconds. Some will be junk the classifier missed.
- Ask the client which supplier calls they actually care about, and add those names to the partner list.
- Check whether any sales caller got a transfer, a booking or a follow-up text. Any of those is a rule that needs tightening.
Each change goes through the same before-and-after check I described in regression testing a live voice agent, because a sharper sales filter is exactly the kind of one-line prompt change that can quietly start declining real customers.
I build production AI voice and chat agents on Retell AI, wired into n8n and GoHighLevel, for clinics, garages, real estate and service businesses. If your phone line needs an agent that can tell a customer from a cold call, see how I build voice agents, read more about my background here, or book a call.