AI transcription hallucinations: what the research found, and how to catch them

Many transcription errors are mishearings: "fifteen" for "fifty", a name spelled wrong. They're annoying, but they're anchored to something that was actually said.

A hallucination is different. It's text with no basis in the audio at all — a phrase or a whole sentence the recogniser produced from nothing. It can read perfectly fluently, there's no half-heard word in your memory of the conversation to make you doubt it, and when it invents something alarming it can do real harm. This post is about what's actually known about it, and what to do.

What the research measured

The study most people are thinking of when they cite numbers is Careless Whisper: Speech-to-Text Hallucination Harms by Allison Koenecke and colleagues, published at ACM FAccT 2024. It tested OpenAI's Whisper, through its hosted API, in spring 2023. What it found, in the paper's own terms:

• The researchers transcribed 13,140 audio segments of people speaking, and 1.4% of the transcriptions contained hallucinations — entire phrases or sentences that didn't exist in the audio in any form. The abstract rounds this to "roughly 1%".
• Of the hallucinations, 38% included explicit harms: invented violence, made-up associations, or false authority.
• The audio came partly from people with aphasia, a language disorder, and partly from a control group. 1.7% of the aphasia segments produced hallucinations, against 1.2% of the control segments.
• Audio that produced hallucinations tended to have a larger share of time without detected speech (29% against 26%) — longer pauses are common in aphasic speech. The authors' hypothesis is that the model is "seeded by noise rather than speech" in those stretches; the study shows the association, not the cause.
• The authors note their count is likely an undercount: the way they detected hallucinations missed ones that repeated identically across runs.

Two further details are often left out, and they matter.

It was a snapshot. Manual review confirmed 187 segments that reliably produced hallucinations. When the researchers re-ran them in December 2023, only 12 of the 187 still did — an improvement they considered likely to be the result of updates to Whisper in late November 2023. Either way, a rate measured on one model in spring 2023 is not a rate for that model today, let alone for any other.

The comparison was narrow. On those 187 segments, the researchers found no comparable hallucinations from Google's, Amazon's, Microsoft's, AssemblyAI's or Rev.ai's services. That's a real finding about those segments and those services at that time. It is not evidence that any of them never hallucinates, and it says nothing about the models they run now.

The numbers that went viral

In October 2024 an Associated Press investigation reported on Whisper being used in hospitals. It included three striking figures, each from one person's own work:

• a University of Michigan researcher studying public meetings found hallucinations in "8 out of every 10" of the transcriptions inspected, before starting to try to improve the model;
• a machine learning engineer found them in about half of more than 100 hours of transcriptions analysed;
• a developer found them in nearly every one of 26,000 transcripts created with Whisper.

These are what individuals found in their own samples. The samples weren't described in enough detail to treat any of them as a rate for the model in general, or to compare them with each other. None of them says "Whisper hallucinates 80% of the time", which is how the first sometimes gets repeated.

The most important detail in that story isn't a number. It's that one widely used medical tool built on Whisper deleted the original audio "for data safety reasons", so its transcripts could never be compared with what was actually said. As a former OpenAI engineer put it to the AP, you can't catch errors if you take away the ground truth.

What this means in practice

The published measurements are of specific models, at specific times, on specific audio. None of them tells you how often the recogniser you're using today will hallucinate on your recordings. Treat a promise of "no hallucinations" as a claim to ask for evidence of, not a fact.

The study points to one place to look, and experience suggests a few more:

• Pauses and stretches without speech. This is the association the study found. Long gaps, slow speakers and quiet passages deserve a second listen.
• Background sound. Developers interviewed by the AP said fabrications tend to occur amid pauses, background sounds or music. That's their observation rather than a measured finding, but it's cheap to act on.
• Fluent text where you remember nothing being said. A tidy sentence sitting in what you remember as a pause, including at the very start or end of a recording, is worth checking.
• Content that doesn't fit. A sudden change of topic, a name nobody mentioned, or an alarming statement that nobody reacted to.

How to catch them

The only reliable check is the one that AP story hinged on: compare the text with the audio.

1. Keep the recording. A transcript without its audio can't be checked. Never let a tool — or your own storage habits — delete the original before you're finished with it.
2. Listen to anything that matters. Every quote you'll publish, every sentence you'll act on, every line that surprises you. With word-level timings, that's a click, not a scrub through the file.
3. Be most careful where speech is sparse. Pauses, the ends of recordings, noisy stretches.
4. Don't trust summaries of unchecked transcripts. A summary of a hallucinated sentence reads just as confidently as a summary of a real one.

What we do, and what we don't claim

txscribe's web app transcribes the audio you upload or record with Google's Speech-to-Text. We haven't measured its hallucination rate, we're not aware of a published measurement for the model we use, and we don't claim it's zero.

What we've built is the checking. In an account, the recording stays alongside the transcript until you delete it (a recording made in the free trial without an account is deleted automatically after 48 hours, so sign in if you need to keep it). Words come back with timestamps from the recogniser, so you can jump from any line of text into the recording and hear it. When you ask a transcript a question, the passages listed under the answer are shown as quotes only when those exact words are in the transcript, each linked to its moment. None of that prevents a hallucination. It makes one quick to find.

For a routine built on that, see how to quote an interview accurately, or try txscribe on one recording without an account.