TL;DR: Speech-to-text is two different engineering problems wearing the same name. Transcribing a fifty minute lecture is a batch problem: nobody is waiting, you can re-run it, and the thing that decides quality is the capture path and what you do with the transcript afterwards. Transcribing a caller on a phone line is a turn-based problem: you get one attempt, the caller hears the result, and latency is part of the product. I have shipped both, an AI note-taking app that records and summarises lectures, and production voice agents that answer real phones. The models overlap. Almost nothing else does. Here is what actually moves accuracy in each, and what carried across.
I did not plan to work on speech twice. LectureNotes AI came first: a note-taking app that records a lecture, summarises the takeaways and builds a clean outline students can revise from. Voice agents came later, and the first time I looked at a Retell call transcript I assumed I already knew this problem.
I did not. The two share a model family and share nothing else.
The same model, two completely different jobs
Long-form transcription is a batch job. A student hits record, sits through an hour, and closes their laptop. The audio arrives as one file. Nobody is standing at the other end of a socket waiting for the next token. If a pass fails, you can run it again with different settings and nobody knows. You can spend three minutes of compute on an hour of audio and the user experiences it as "it was ready when I looked".
A voice agent is the opposite in every dimension. Audio arrives as a stream, the useful unit is a three to eight second turn, and the transcript is consumed immediately by a model that is about to say something out loud. There is no second pass. There is no "run it again with a bigger model". The caller is on the line, and every millisecond you spend is silence they are listening to.
Once you see it that way, the design decisions stop being about which STT provider is best and start being about which failure you can afford.
In batch, the capture path beats the model
The single biggest accuracy improvement on the note-taking side had nothing to do with the transcription model. It was recording conditions.
A lecture recording is a phone on a desk, six rows back, in a room with hard walls and an air conditioner. Sometimes the phone is in a bag. Sometimes the student is at the back. The lecturer walks. Other students talk. Compared to that, the difference between two good STT models is noise.
What helped was unglamorous: handling the recording session properly so it survives the screen locking and the app going to background, being honest in the interface about what a bad recording will produce, and getting the audio format and sample rate right at the source rather than repairing it later. If you upsample rubbish you get expensive rubbish.
That lesson is the same one I hit later in clinical eye-tracking work: the instrument comes before the analysis. If your capture is compromised, no downstream cleverness recovers it, and pretending otherwise is how you ship confident nonsense.
The words that matter are not the words in the accuracy score
Word error rate treats every word as equal. Users do not.
A lecture transcript that is broadly correct but mangles the lecturer's specific terminology is useless, because the terminology is the entire reason the student is revising. Course jargon, people's names, formula names and abbreviations are exactly the tokens with the least redundancy in the surrounding sentence, so context cannot repair them. Meanwhile a transcript that drops filler words and tidies false starts is often better than a faithful one.
So the useful evaluation was never overall accuracy. It was: did the terms survive? Feeding known domain vocabulary in as a bias or prompt hint bought more perceived quality than any model swap, because it targeted the small set of tokens carrying all the meaning.
That is the exact same failure shape I later wrote about in capturing caller details on a phone line. Identifiers have no redundancy. Nothing around a postcode, a registration number or an email address tells you what it should have been.
What you do after the transcript decides the product
For long-form, the transcript is not the deliverable. The summary and the outline are. And that step is where quality is genuinely won.
Three things mattered more than I expected:
Chunk on the audio, not on the character count. Cutting a transcript into fixed windows regularly slices a definition in half, and the summary then confidently loses it. Chunking around natural pauses, with overlap between chunks, produced noticeably better output.
Keep timestamps all the way through. If a student cannot jump from a summary line back to the moment it came from, they cannot check it, and if they cannot check it they will not trust it. Traceability is a feature, not plumbing.
Structure is a product decision, not a prompt afterthought. Students did not want prose. They wanted something scannable: headings, short lines, the definitions pulled out. An excellent essay-shaped summary tested worse than a mediocre outline-shaped one.
I use the same instinct on voice work now. The transcript is raw signal. Everything the client actually cares about, the booking, the CRM record, the outcome label, is a structured extraction sitting on top of it.
On a phone line, you are transcribing through a straw
Then there is the call.
Telephony audio is narrowband. It is not a lightly degraded version of good audio, it is audio with the top of the spectrum removed, and the frequencies it removes are the ones that distinguish consonants. That is why f and s collapse into each other, why m and n become a coin toss, and why the digits five and nine are a recurring problem. This is not a model failing. It is information that never reached the model.
On top of that: mobile callers in cars, hands-free, on speaker, in a busy shop. Regional accents that a US-tuned recognition setting handles badly, which is one of the things that breaks when a US-built agent starts answering a UK phone line.
Because the audio is worse and the stakes are higher, the fix has to move out of the STT layer entirely. You cannot make the line better. You can design a conversation that is robust to mishearing: confirm identifiers in chunks as you take them, validate shape downstream in automation rather than hoping the prompt caught it, and build a repair ladder with an exit so a bad capture becomes a text message or a transfer rather than a loop.
Latency is only a problem in one of these
In batch, latency is a scheduling detail. In a call, it is the product.
Streaming STT adds cost at exactly the point you are most sensitive: endpointing has to decide when the caller has actually stopped talking, and that decision is a direct trade against being interrupted or leaving dead air. Set it aggressively and the agent talks over people. Set it conservatively and the agent feels slow. It is one link in the chain I broke down in the voice agent latency budget, and it is the link you tune per moment rather than globally, because taking a phone number and hearing "yes" want opposite settings.
Nothing in a lecture app has an equivalent to that. Nobody has ever complained that a summary arrived four hundred milliseconds late.
The failures are visible in one and silent in the other
This is the difference I underrate least now.
A bad lecture transcript is obvious. The user reads it, sees the mess, and either re-records or complains. The feedback loop is short and self-correcting.
A bad phone capture is silent. The call sounds fine. The caller is polite. The agent confirms something plausible, the automation writes a record, and the failure only surfaces days later when an email bounces or a customer does not turn up. Nobody watched it happen. That is why I score voice agents per field rather than per call, and why I insist on storing the raw transcript and audio alongside the extracted result: when the structured value is wrong, the only way to find out why is to go back to the signal.
If you are choosing between them today
A quick heuristic, since I get asked this by founders building both kinds of thing:
- If nobody is waiting, buy accuracy. Use the larger model, do a second pass, feed in domain vocabulary, spend the seconds. Cost per hour of audio is your constraint, not latency.
- If someone is on the line, buy predictability. Consistent, low, boring latency beats a slightly better transcript, because the caller experiences the pause and never sees the transcript.
- Either way, evaluate on the tokens that carry meaning, not on an aggregate score. Pick the twenty terms or five fields that matter and measure those.
- Store the raw signal. Transcript, audio, timestamps. Every interesting debugging session I have had in either product started by going back to the recording.
The uncomfortable conclusion after shipping both is that the model was rarely the interesting variable. Capture conditions, chunking, structured extraction and honest evaluation did more for both products than any provider comparison, and those are all things you own rather than things you buy.
If you want the voice agent side of this in more depth, how I build voice agents on Retell covers the stack, and writing prompts for the ear covers the half of the problem that happens after the transcript is correct.
I build production AI voice agents and the automation, CRM and telephony layer behind them for founders across the US, UK and Europe. More about my work here, or book a call.