When "auto-detect language" quietly translates instead of transcribing
·9 min read
Most transcription failures announce themselves. A name comes out mangled, a technical term becomes nonsense, a stretch of crosstalk turns into word salad. You read it, you wince, you fix it.
There's one failure that doesn't announce itself, and it's the one worth knowing about. In recordings where people switch between languages mid-conversation, a transcription model set to "auto-detect" will sometimes stop transcribing and start translating — handing you a fluent, grammatical, entirely plausible English sentence for something that was said in Mandarin. Or the reverse.
Nothing looks wrong. That's the problem.
What it looks like
A product meeting, two Chinese speakers and one American, the kind of call where people switch languages by sentence depending on who they're addressing. Set the language to auto and you can get a transcript where a Mandarin remark appears in clean English, sitting between two English sentences, with correct punctuation and a sensible timestamp.
If you weren't in the room, you will never know. The transcript is coherent. It's the right meaning, more or less. It's in the right place. The only thing wrong with it is that it isn't what anybody said — and if your job involves quoting people, "isn't what anybody said" is the whole ballgame.
We saw this once, on a real bilingual recording, on 16 September 2026. We have not been able to reproduce it since — not because it stopped happening, but because we lost access to the credentials needed to run the test again. So: one observation, dated, unconfirmed. We are telling you anyway, because the mechanism underneath it is not in doubt even if the frequency is.
Why a transcriber would translate
It isn't a bug in the ordinary sense. It falls out of how modern multilingual speech models are built.
The older generation of speech recognition was a pipeline: acoustic model, pronunciation dictionary, language model, one language at a time. It couldn't translate if you asked it to.
The current generation is a single sequence model. Audio goes in, text comes out, and the model is steered by control tokens that say what task to do and in what language. Crucially, these systems are trained on more than one task at once — transcription and speech translation share the same weights, because training them together makes both better. Whisper's own task tokens are named transcribe and translate for exactly this reason. The machinery to translate is right there, one token away, by design.
Now set the language to auto. You've asked the model to infer the control token rather than be told it. It listens to some audio, decides "this is an English recording", and locks that in. Then somebody speaks Mandarin. The model has already committed to producing English text. It has a perfectly good path to doing that for Mandarin input — the translation path — and it takes it.
The decoder isn't malfunctioning. It's doing the most fluent thing available given a premise it was allowed to choose for itself. The failure is that nobody told it the premise was supposed to hold for the entire three-hour recording.
Why it hides
Every other transcription error degrades the text. This one improves it.
Proofreading catches errors that read wrong. Your eye stops on "the patient was prescribed amoxicillin" becoming "the patient was prescribed a mock suspension" because the sentence is broken. A clean translation breaks nothing. It survives a read-through, a spellcheck, and a skim by someone who was in the meeting but doesn't remember which language that particular sentence was in.
It also survives the summary. Ask for a summary of a transcript with silent translations in it, and you get a summary that's arguably accurate as to meaning and completely unusable as to attribution. The quotes you pull from it will be quotes nobody said.
This is the concrete reason we tie every summary line and every answer back to a timestamp in the audio. A claim you can hear is a claim you can catch. A claim floating free in a notes document is one you have to take on faith, and this is precisely the situation where faith is misplaced.
What to do instead
Name the language. If most of the recording is Mandarin with English terms sprinkled through it, set it to Mandarin. A model told which language it's in will transcribe foreign words as heard rather than reaching for the translation path. You'll get some borrowed English words rendered phonetically, which is a visible error you can fix in three seconds — infinitely better than an invisible one you can't find.
The rule of thumb: auto-detect is for recordings in one language you don't know in advance, not for recordings in two languages you do. It answers "which language is this?" — a question with one answer. A code-switched conversation doesn't have one answer, so the question is wrong before the model ever gets it.
Spot-check at the switches. If you know the conversation moved between languages, click through to those moments and listen. In txscribe every line is clickable and plays the audio behind it, which is the fastest version of this check we know how to build. Ten seconds of listening at three switch points is usually enough to tell you whether the transcript is honest.
Be suspicious of prose that's too clean. Real speech in a second language has hesitations, restarts, and grammar that wobbles. A transcript where a non-native speaker's English is flawless and idiomatic for one paragraph and normal everywhere else is worth a listen.
Why you can't just detect it
The obvious engineering answer is to detect the failure and warn the user. We built that, and then we shelved it, and the reason is the most useful thing we learned.
The detector looked for gaps in word-level timestamp coverage: if a stretch of audio produced no words, something was lost there, so flag it. It worked. It had seventeen tests, and every one of them could make it fail.
But a gap detector can only catch a failure that leaves a gap. That is a property of the detector, not a bug in it, and it is true regardless of which engine you point it at. Dropping a language leaves a hole. Translating one does not — the translated text occupies the timeline exactly as transcribed text would, every second accounted for, every word timed. Neither does a model that simply decodes the audio as if it were a language it isn't: you get fluent-looking nonsense, at full coverage.
We could not rule out either of those shapes, and in one case we could read the answer in source. The on-device pipeline behind our desktop app runs Whisper through transformers.js. In the version we were on — 4.3.0, read on 20 September 2026 — src/models/whisper/modeling_whisper.js builds the decoder's prefix tokens like this:
if (generation_config.is_multilingual) {
if (!language) {
// TODO: Implement language detection
logger.warn('No language specified - defaulting to English (en).');
language = 'en';
}
// ...
}The outer condition matters, so read it too: this applies to a multilingual checkpoint, which is the only kind where "automatic" means anything and is what an app handling more than one language runs. Within that case, leaving the language unset does not trigger detection. There is no detection to trigger.
Note what it does instead, because this is the part worth taking away. It is not silent. It logs a warning that says exactly what is about to go wrong, in plain English, and then does it anyway. The library is not hiding anything — the TODO is right there, and so is the warning.
The warning was just addressed to someone who wasn't listening. It goes to a log, and in production nobody reads the log; the person affected is holding a transcript that looks complete. That gap — between where a system reports a problem and where the consequence lands — is the more general version of everything else in this post, and most teams have one.
Two caveats on that snippet, in the spirit of the thing. It is version-specific: if upstream implements detection, the quote above stops being true, so check the version you actually have rather than trusting this paragraph's date. And line numbers move, so search for the TODO rather than jumping to a line.
What we did about our half of it
Ours was the fixable half, so this is no longer how our desktop app behaves. The ?? 'en' fallback is gone. It now asks the model directly: one forward pass with nothing but <|startoftranscript|> in the decoder puts the whole language distribution in the first position's logits, and a softmax over just the language tokens gives a winner and a confidence. It probes up to twelve points spread across the file rather than only the opening — a meeting that starts in English and turns doesn't announce itself in the first thirty seconds — and below 50% confidence it refuses to run and asks you to pick the language yourself.
Refusing is the point. Guessing English is a decision the software makes on your behalf and never mentions again; stopping to ask is one you get to make.
A detector aimed at gaps, then, would have reported a clean bill of health on at least one real failure mode and possibly three. So we took it out of the pipeline and wrote down what would let us put it back: run a recording that is half English and half Mandarin through the cloud engine's auto mode, and confirm that content is genuinely discarded rather than translated or hallucinated.
The phrasing we first reached for was "a check that cannot fail is not a check", and that is wrong in an instructive way. The check could fail; seventeen tests made it. It just could never fail for the thing it was built to catch, because the signal it measured and the damage it was aimed at are not the same shape. That is the harder version of the problem, and the one worth watching for: not a fake check, but a real check pointed at the wrong quantity.
Where we are on it
We don't have a complete fix, and we're not going to claim one. Forcing a language stops the silent translation, but it costs you accuracy on the genuinely mixed passages. Running detection per segment instead of per file helps and introduces its own instability at the boundaries — the same seam problem that makes speaker labels drift on long recordings.
What we can do is make the failure findable. Every word carries a timestamp, every line plays, and an answer to a question about the recording cites the second it came from. When the machine invents something fluent, the audio is still there to contradict it.
That's the whole posture, really. We don't think the useful claim is that our transcripts are never wrong. It's that when one is wrong, you can find out in two seconds instead of finding out in print.
Corrected 20 September 2026: this post said every summary points back to the second it came from. Answers to questions do; summaries don't.