TL;DR: You do not need an ML team to test what your AI product says. You need three things: a small, named set of real inputs that represent your users (thirty to fifty is plenty to start), a written list of what a good output must and must never do for each one, and a habit of running that set before every prompt or model change. Check the structural rules with code, check the judgement calls with a model as a grader only where you have confirmed it agrees with you, and read a handful of outputs yourself every time. The goal is not a score. It is knowing, before your users do, that a change made something worse.
Every AI product I have worked on hit the same moment. Someone changes a prompt, or swaps the model for a cheaper one, or adds a field to the output, and it looks fine on the two examples they tried. A week later a user screenshots something that is clearly wrong, and nobody can say when it started.
I have seen that moment from a few angles. At LectureNotes AI the output was a summary, takeaways and an outline built from a recorded lecture. At Lifemaxxing AI it is a personalised plan. In client work at Fortell AI and Tested Media it is a voice or chat agent's behaviour on a real call. The products are different. The testing problem is the same, and the fix is smaller than most founders expect.
Why "it looked fine when I tried it" is not a test
Normal software fails loudly. A broken function throws, a test goes red, the build stops. LLM output fails politely. It still returns well-formed text, it still sounds confident, and it is subtly worse: a definition dropped from a summary, a plan that ignores the constraint the user gave, an agent that now asks for the phone number twice.
Three things make this worse in a small product:
- Changes are cheap, so they happen constantly. Editing a prompt takes thirty seconds and does not feel like a deploy. It is one.
- Nobody tries the same inputs twice. The founder tests with whatever comes to mind, which is never the awkward case from last Tuesday.
- Regressions hide in the long tail. The common case usually survives. It is the short lecture, the user with an unusual goal, the caller with a strong accent that quietly breaks.
The answer is not a research-grade evaluation framework. It is the discipline of running the same inputs every time and writing down what "good" means before you look at the output.
Step one: build a small set from real inputs
Start with a test set of real inputs, not invented ones. Invented examples are the ones you already know how to handle, which is exactly why they are useless for catching regressions.
Where mine have come from:
- Things users actually sent. Real lecture recordings of different lengths and quality, real goals people typed into an onboarding flow, real call transcripts. Strip or replace anything personal before it goes into a test set.
- Every complaint. Each time a user reports a bad output, that input goes into the set permanently. This is the single highest value habit. A bug you have fixed once should never come back silently.
- The edges on purpose. Very short and very long inputs, empty or near-empty inputs, inputs in a different language, inputs that try to push the model off its job.
Thirty to fifty cases is enough to start. More is not better if nobody looks at the results. I would rather have forty cases that get run before every change than four hundred that get run once.
Give each case a name a human understands, like "45 minute lecture, poor audio, lots of jargon" or "goal with a hard time constraint". When a case fails, the name should tell you what kind of thing broke before you even open it.
Step two: write down what good looks like, per case
This is the step people skip, and it is the step that makes everything else work. For each case, write a short list of things the output must do and must never do. Plain sentences, not scores.
For a lecture summary that looked like:
- The outline uses headings and short lines, not paragraphs.
- The three named terms from the lecture appear, spelled correctly.
- Nothing appears in the summary that was not said in the lecture.
For a personalised plan:
- The plan respects the time the user said they have each day.
- It stays inside the fixed structure the app expects.
- It does not give medical or diet advice the product is not allowed to give.
For a voice agent, the same idea becomes a scenario with a required outcome, which I go into in detail in how I regression test voice agents.
Writing these down forces the real product decisions into the open. At LectureNotes the most useful thing I learned was that an outline-shaped summary beat a beautifully written essay-shaped one. That is not something a generic quality score would ever have told me. It came from deciding what the student actually needed and then checking for it.
Step three: check what you can with plain code
A surprising amount of "is this output good" is checkable with ordinary code, no model required. These checks are fast, free and never disagree with themselves, so they should do as much of the work as possible.
Structure
If the output is supposed to be JSON with certain fields, parse it and check the fields. If it is supposed to be an outline, check that it has headings. If a plan should have seven days, count them. This is why I push hard for structured output in AI products, the argument I make in why personalised AI apps should not be a chatbot. Structured output is testable output.
Required and forbidden content
If a lecture mentions specific terms, check they survive. If a plan must never mention a banned topic, search for it. If an agent's final summary must include a callback number, check the field is present and shaped like a phone number. Crude string checks catch a lot of real regressions.
Length and cost
Track output length and tokens per case. A change that doubles the length of every summary is a product change and a cost change at the same time. I covered why that matters in AI app unit economics, and the test set is the cheapest place to notice it.
Step four: use a model as a grader, carefully
Some rules cannot be checked with code. "Nothing in the summary was not said in the lecture" or "the agent stayed polite with an upset caller" are judgement calls. Using a second model call to grade these is reasonable, with two conditions.
Ask narrow, yes or no questions. "Rate this summary out of ten" gives you a number that drifts and means nothing. "Does the summary contain any claim that is not supported by the transcript below? Answer yes or no and quote the claim" gives you something you can act on and check.
Confirm the grader agrees with you before you trust it. Grade twenty outputs yourself, run the model grader on the same twenty, and compare. Where it disagrees with you, fix the grading question or keep that check manual. A grader you have never checked is just a second, unreviewed prompt making decisions about your product.
I also keep the grader on a different model from the one producing the output where I can, and I keep its prompt under version control exactly like the product prompt. If the grading question changes, old results are no longer comparable, and you want to know that.
Step five: read outputs yourself, every time
Automated checks tell you something broke. Reading tells you something got worse in a way you had not thought to check. Before shipping a change, I read a handful of outputs side by side with the previous version, starting with the cases that changed the most.
This is where most of my real findings come from. A summary that passes every check but has lost its rhythm. A plan that is technically correct but reads like a form letter. An agent that now says "absolutely" in every turn. None of these fail a rule, and all of them are things a user notices on day two.
At Fortell AI the pipeline I built with Claude Code and Comet browser automation made it fast to build and run agents against test scenarios. Fast tooling is what makes the human reading affordable. It does not replace it.
When to run it
Run the full set before any of these, without exception:
- A prompt edit, however small. Small edits are the ones nobody tests.
- A model change, including a version bump from the same provider. "Same model, newer version" can change tone, length and how strictly instructions are followed. If you are weighing models, how I choose the LLM for a voice agent has the trade-offs.
- A change to the input pipeline. At LectureNotes the capture and transcription side mattered as much as the prompt, which I wrote about in what a lecture recorder and a phone agent taught me about speech-to-text. Changing chunking or audio handling changes the output just as much as changing the prompt.
- A change to the output format, since downstream code and your app's screens depend on it.
Keep the results of each run, even as a simple file per run with the date and what changed. When a user reports something odd three weeks later, being able to say "this case passed on the 3rd and failed on the 10th, after the model change" turns a mystery into a ten minute fix.
The short version
Take thirty to fifty real inputs, including every past complaint and a few deliberate edge cases. Write down what each output must and must never do. Check structure and required content with code, use a model grader only for narrow questions where you have confirmed it agrees with you, and read a handful of outputs yourself before every change. It takes an afternoon to set up, and it is the difference between finding a regression yourself and finding it in a user's screenshot.
I build AI products, voice and chat agents, and mobile and web apps for founders across the US, UK and Europe, including my own AI apps with real users. If you are shipping an AI feature and want a way to change it without holding your breath, more about me is here, or book a call and we can set up a test set for your product.