When "auto-detect language" quietly translates instead of transcribing
·9 min read
Most transcription failures announce themselves. A name comes out mangled, a technical term becomes nonsense, a stretch of crosstalk turns into word salad. You read it, you wince, you fix it.
There's one failure that doesn't announce itself, and it's the one worth knowing about. In recordings where people switch between languages mid-conversation, a transcription model set to "auto-detect" will sometimes stop transcribing and start translating — handing you a fluent, grammatical, entirely plausible English sentence for something that was said in Mandarin. Or the reverse.
Nothing looks wrong. That's the problem.
What it looks like
A product meeting, two Chinese speakers and one American, the kind of call where people switch languages by sentence depending on who they're addressing. Set the language to auto and you can get a transcript where a Mandarin remark appears in clean English, sitting between two English sentences, with correct punctuation and a sensible timestamp.
If you weren't in the room, you will never know. The transcript is coherent. It's the right meaning, more or less. It's in the right place. The only thing wrong with it is that it isn't what anybody said — and if your job involves quoting people, "isn't what anybody said" is the whole ballgame.
We saw this once, on a real bilingual recording, on 16 September 2026. We have not been able to reproduce it since — not because it stopped happening, but because we lost access to the credentials needed to run the test again. So: one observation, dated, unconfirmed. We are telling you anyway, because the way these models are built makes it entirely plausible, even though we can't say how often it happens.
Why a transcriber would translate
It isn't a bug in the ordinary sense. It falls out of how modern multilingual speech models are built.
The older generation of speech recognition was a pipeline: acoustic model, pronunciation dictionary, language model, one language at a time. It couldn't translate if you asked it to.
The current generation is a single sequence model. Audio goes in, text comes out, and the model is steered by control tokens that say what task to do and in what language. Crucially, these systems are trained on more than one task at once — transcription and speech translation share the same weights, because training them together makes both better. Whisper's own task tokens are named transcribe and translate for exactly this reason. The machinery to translate is right there, one token away, by design.
Now set the language to auto. You've asked the model to infer the control token rather than be told it. What we saw fits a model that decided a stretch of audio was English and then met Mandarin inside that stretch: already producing English text, it has a perfectly good path to doing that for Mandarin input — the translation path. We can't see inside the recogniser we use, so take that as the likeliest explanation, not a proven one.
If that is what happened, nothing malfunctioned. The model did the most fluent thing available given a premise it was allowed to choose for itself — a premise chosen for a stretch of audio that was only true of some of the sentences in it.
Why it hides
Every other transcription error degrades the text. This one improves it.
Proofreading catches errors that read wrong. Your eye stops on "the patient was prescribed amoxicillin" becoming "the patient was prescribed a mock suspension" because the sentence is broken. A clean translation breaks nothing. It survives a read-through, a spellcheck, and a skim by someone who was in the meeting but doesn't remember which language that particular sentence was in.
It also survives the summary. Ask for a summary of a transcript with silent translations in it, and you get a summary that's arguably accurate as to meaning and completely unusable as to attribution. The quotes you pull from it will be quotes nobody said.
This is the concrete reason answers in chat cite the moments they came from, as timestamps in the audio. A claim you can hear is a claim you can catch. A claim floating free in a notes document is one you have to take on faith, and this is precisely the situation where faith is misplaced.
What to do instead
Name the language. If most of the recording is Mandarin with English terms sprinkled through it, set it to Mandarin — then listen back at the places where people switched, because we can't promise what the recogniser does with the English once it has been told Mandarin. If borrowed English words come back rendered phonetically, that is a visible error you can fix in three seconds — infinitely better than an invisible one you can't find.
The rule of thumb: auto-detect is for recordings in one language you don't know in advance, not for recordings in two languages you do. It has to answer "which language is this?" for each stretch of audio, and a stretch where people switch languages doesn't have one answer — so the question is wrong before the model ever gets it.
Spot-check at the switches. If you know the conversation moved between languages, click through to those moments and listen. In txscribe every line is clickable and plays the audio behind it, which is the fastest version of this check we know how to build. Ten seconds of listening at three switch points is usually enough to tell you whether the transcript is honest.
Be suspicious of prose that's too clean. Real speech in a second language has hesitations, restarts, and grammar that wobbles. A transcript where a non-native speaker's English is flawless and idiomatic for one paragraph and normal everywhere else is worth a listen.
Why you can't just detect it
The obvious engineering answer is to detect the failure and warn the user. We built that, and then we shelved it, and the reason is the most useful thing we learned.
The detector looked for gaps in word-level timestamp coverage: if a stretch of audio produced no words, something was lost there, so flag it. It worked. It had seventeen tests, and every one of them could make it fail.
But a gap detector can only catch a failure that leaves a gap. That is a property of the detector, not a bug in it, and it is true regardless of which engine you point it at. Dropping a language leaves a hole. Translating one does not — the translated text occupies the timeline exactly as transcribed text would, every second accounted for, every word timed. Neither does a model that simply decodes the audio as if it were a language it isn't: you get fluent-looking nonsense, at full coverage.
We could not rule out either of those shapes, and in one case we could read the answer in source. The on-device pipeline behind our desktop app runs Whisper through transformers.js. In the version we were on — 4.3.0, read on 20 September 2026 — src/models/whisper/modeling_whisper.js builds the decoder's prefix tokens like this:
if (generation_config.is_multilingual) {
if (!language) {
// TODO: Implement language detection
logger.warn('No language specified - defaulting to English (en).');
language = 'en';
}
// ...
}The outer condition matters, so read it too: this applies to a multilingual checkpoint, which is the only kind where "automatic" means anything and is what an app handling more than one language runs. Within that case, leaving the language unset does not trigger detection. There is no detection to trigger.
Note what it does instead, because this is the part worth taking away. It is not silent. It logs a warning that says exactly what is about to go wrong, in plain English, and then does it anyway. The library is not hiding anything — the TODO is right there, and so is the warning.
The warning was just addressed to someone who wasn't listening. It goes to a log, and in production nobody reads the log; the person affected is holding a transcript that looks complete. That gap — between where a system reports a problem and where the consequence lands — is the more general version of everything else in this post, and most teams have one.
Two caveats on that snippet, in the spirit of the thing. It is version-specific: if upstream implements detection, the quote above stops being true, so check the version you actually have rather than trusting this paragraph's date. And line numbers move, so search for the TODO rather than jumping to a line.
What we did about our half of it
Ours was the fixable half, so this is no longer how our desktop app behaves. The ?? 'en' fallback is gone. It now asks the model directly: one forward pass with nothing but <|startoftranscript|> in the decoder puts the whole language distribution in the first position's logits, and a softmax over just the language tokens gives a winner and a confidence. It probes up to twelve points spread across the file rather than only the opening — a meeting that starts in English and turns doesn't announce itself in the first thirty seconds — and below 50% confidence it refuses to run and asks you to pick the language yourself.
Refusing is the point. Guessing English is a decision the software makes on your behalf and never mentions again; stopping to ask is one you get to make.
A detector aimed at gaps, then, would have reported a clean bill of health on at least one real failure mode and possibly three. So we took it out of the pipeline and wrote down what would let us put it back: run a recording that is half English and half Mandarin through the cloud engine's auto mode, and confirm that content is genuinely discarded rather than translated or hallucinated.
The phrasing we first reached for was "a check that cannot fail is not a check", and that is wrong in an instructive way. The check could fail; seventeen tests made it. It just could never fail for the thing it was built to catch, because the signal it measured and the damage it was aimed at are not the same shape. That is the harder version of the problem, and the one worth watching for: not a fake check, but a real check pointed at the wrong quantity.
Where we are on it
We don't have a complete fix, and we're not going to claim one. Forcing a language is the best defence we know against the silent translation, not a proven cure, and it costs you accuracy on the genuinely mixed passages. Running detection per segment instead of per file helps and introduces its own instability at the boundaries — the same seam problem that makes speaker labels drift on long recordings.
What we can do is make the failure findable. Every word carries a timestamp, every line plays, and an answer to a question about the recording cites the second it came from. When the machine invents something fluent, the audio is still there to contradict it.
That's the whole posture, really. We don't think the useful claim is that our transcripts are never wrong. It's that when one is wrong, you can find out in two seconds instead of finding out in print.
Corrected 20 September 2026: this post said every summary points back to the second it came from. Answers to questions do; summaries don't.