No accuracy number from us. Here's the test instead.

7 November 2026·15 min read

We've said before that we won't put an accuracy number on the site, and why: the number is measured on audio that isn't yours, by a normaliser nobody states, against a reference that wobbles, and it weights a missed "um" the same as a missed "not".

That doesn't mean we don't measure. We have to. We run on cloud recognisers, there is more than one to choose from, and the choice has to be made on something better than a vendor's own page. So we built a test, and in October we ran two candidate engines through it.

What follows is everything you need to rebuild the test and score with it: where the audio comes from and the code that rebuilds it, the term lists, the normaliser, the scoring, the rules we run engines by, and what running it taught us about the method. What's missing is the score. That's deliberate. A number travels without its conditions; the test is the conditions. Publish those and anyone can re-run it against this month's models, on their own candidates, including us. Publish the number and all you can do is quote it.

The audio

The open Portuguese and Spanish corpora with human transcripts that we could use come cut into utterances, five to twenty seconds each. A class or a meeting runs for an hour, and a five-second utterance can't show how errors build up or carry on across one: whether a recogniser recovers after a stretch of students talking over each other, say, or writes a passage out twice. So we rebuilt longer stretches from them: four of about fifteen minutes and two of seven and a half.

Take one recording, take its segments in time order, join them back together with a quarter-second of silence after each, and join the segments' transcripts in the same order for the reference. The silence is there to keep one utterance's last word from running into the next one's first. It doesn't always work, and the original pauses between utterances are gone; both are in the list of weaknesses below.

Six clips, 72 minutes in all:

pt  NURC-SP EF_057          lecture: Shakespeare's tragedies       14.4 min, from 120 s in
pt  NURC-SP EF_154          architecture seminar, chaired          14.4 min, from 120 s in
pt  NURC-SP EF_106          income-tax law class, student Q&A      first 449 s of 15 min from 120 s
pt  NURC-SP EF_187          educational assessment class           first 449 s of 15 min from 120 s
es  TEDx Spanish F_032      stage talk, close mic                  14.4 min
es  Tele con Ciencia F_21   science radio interview, studio/phone  14.4 min

The sources, all on Hugging Face:

NURC-SP, nilc-nlp/CORAA-NURC-SP-Audio-Corpus, test split, the "lecture and talks" genre. Real lectures recorded in São Paulo on analogue tape decades ago; the transcripts were made or checked by people. Licence CC BY-NC-ND 4.0.
TEDx Spanish, ciempiess/tedx_spanish. Stage talks, close mic, transcribed by native-speaker students. CC BY-NC-ND 4.0.
Tele con Ciencia, ciempiess/tele_con_ciencia. A CONACYT science radio programme with expert transcripts. CC BY-SA 4.0.

Two things about that list are choices, not accidents.

The long clips stop at 14.4 minutes because at least one cloud engine caps a single request at fifteen minutes once you ask for word timestamps or speaker labels. Staying under it means every engine sees each file whole. Nothing gets sliced, so you are scoring the recogniser and not your own stitching code.

The two 449-second windows are the same kind of build cut short, because the first paid cloud check in this series had a budget of thirty minutes for four files. We kept the cut so later runs would score the same audio. Once, a later run's reference for one window came out different, and we couldn't tell why; the checkpoints below are what came of it. The cut lands inside one of the inserted gaps, so the reference is exactly the segments you can hear and not half of one.

And the licences are why you'll be rebuilding this rather than downloading it. NURC and TEDx are no-derivatives, non-commercial. We can tell you the recipe exactly; we can't ship you the WAVs. Fetch the shards, about 5 GB in all:

H=https://huggingface.co/datasets
mkdir -p dl/nurc dl/tedx dl/tlcc
for i in 0 1 2; do curl -L -o dl/nurc/test-0000$i.parquet $H/nilc-nlp/CORAA-NURC-SP-Audio-Corpus/resolve/main/data/original/test-0000$i.parquet; done
for i in 0 1 2 3; do curl -L -o dl/tedx/train-0000$i-of-00004.parquet $H/ciempiess/tedx_spanish/resolve/main/tedx_spanish/train-0000$i-of-00004.parquet; done
for i in 0 1 2 3 4; do curl -L -o dl/tlcc/train-0000$i-of-00005.parquet $H/ciempiess/tele_con_ciencia/resolve/main/tele_con_ciencia/train-0000$i-of-00005.parquet; done

Then build. This is the whole build, as one file:

import glob, io, os
import numpy as np, soundfile as sf, pyarrow.parquet as pq

def segments(shards, recordings, nurc):
    """Each recording's segments in time order: (position, transcript, audio bytes)."""
    found = {r: [] for r in recordings}
    for path in shards:
        f = pq.ParquetFile(path)
        for g in range(f.num_row_groups):
            d = f.read_row_group(g).to_pydict()
            for i in range(len(d['audio'])):
                rec = d['audio_name'][i] if nurc else d['speaker_id'][i]
                if rec in found:
                    pos = float(d['start_time'][i]) if nurc else int(d['audio_id'][i].rsplit('_', 1)[1])
                    text = d['text'][i] if nurc else d['normalized_text'][i]
                    found[rec].append((pos, text, d['audio'][i]['bytes']))
    return {r: sorted(s, key=lambda x: x[0]) for r, s in found.items()}

def build(clip, segs, start, minutes):
    """Segments from `start` on, 0.25 s of silence after each, until the next would pass `minutes`."""
    gap = np.zeros(4000, dtype=np.float32)
    audio, texts, total = [], [], 0.0
    for pos, text, data in segs:
        if pos < start:
            continue
        a, sr = sf.read(io.BytesIO(data), dtype='float32')
        if a.ndim > 1:
            a = a.mean(axis=1)
        assert sr == 16000
        if total + len(a) / sr > minutes * 60:
            break
        audio += [a, gap]
        texts.append(text.strip())
        total += len(a) / sr + 0.25
    sf.write(f'wav/{clip}.wav', np.concatenate(audio), 16000, subtype='PCM_16')
    open(f'ref/{clip}.txt', 'w').write(' '.join(texts) + '\n')
    return texts

def window(clip, texts, seconds=449):
    """The first `seconds`, cut 2,000 samples into an inserted gap: exactly the segments heard."""
    x, sr = sf.read(f'wav/{clip}.wav', dtype='int16')
    z = np.diff(np.concatenate([[0], (x == 0).astype(np.int8), [0]]))
    gaps = [a for a, b in zip(np.where(z == 1)[0], np.where(z == -1)[0]) if b - a >= 3800]
    assert len(gaps) == len(texts)  # one run of silence per segment, none inside one
    k = max(i for i, a in enumerate(gaps) if a + 2000 <= seconds * sr)
    sf.write(f'wav/{clip}_w75.wav', x[:gaps[k] + 2000], sr, subtype='PCM_16')
    open(f'ref/{clip}_w75.txt', 'w').write(' '.join(texts[:k + 1]) + '\n')

os.makedirs('wav', exist_ok=True)
os.makedirs('ref', exist_ok=True)
nurc = segments(sorted(glob.glob('dl/nurc/*.parquet')), ['SP_EF_057', 'SP_EF_154', 'SP_EF_106', 'SP_EF_187'], True)
build('pt_EF_057', nurc['SP_EF_057'], 120, 14.4)
build('pt_EF_154', nurc['SP_EF_154'], 120, 14.4)
window('pt_EF_106', build('pt_EF_106', nurc['SP_EF_106'], 120, 15))
window('pt_EF_187', build('pt_EF_187', nurc['SP_EF_187'], 120, 15))
build('es_TEDX_F_032', segments(sorted(glob.glob('dl/tedx/*.parquet')), ['F_032'], False)['F_032'], 0, 14.4)
build('es_TLCC_F_21', segments(sorted(glob.glob('dl/tlcc/*.parquet')), ['F_21'], False)['F_21'], 0, 14.4)

Open each ref/<clip>.txt and read a paragraph. It should be continuous speech; if a segment is out of order, the corpus's timestamps are, and you'll have seen it before it costs you anything. Then check the two windows against what our first run recorded and a rebuild this month matched exactly. The word counts are after the normaliser below:

pt_EF_106_w75   76 segments   cut at 447.555 s   842 reference words
pt_EF_187_w75   55 segments   cut at 448.403 s   1,169 reference words

If yours differ, the corpus has changed or the build has. Find out which before any engine runs.

The term lists

Word error rate counts every word the same, which is the main thing wrong with it. A transcript can score well and still get wrong the few words a reader needed, so beside WER the test counts a list of subject terms for each recording.

For each recording we picked fourteen to seventeen subject terms by hand from the reference. An occurrence counts as correct only if every word of the term lines up as an exact match in the alignment; "arquitetura" transcribed as "arquitectura" is a miss, and so is "faculdade de arquitetura" with the middle word dropped. Terms the reference itself misspells were left out, because a recogniser shouldn't be marked down for spelling a word correctly.

The lists, verbatim:

EF_057  shakespeare, tragédia, tragédias, vingança, julieta, senequiana, senequiano,
        maturidade, moralidade, consciência, vingador, danação, bastardo, regeneração,
        inglaterra, psicológica, cronológica
EF_106  abatimento, abatimentos, comprovação, tributável, prospecção, imposto, dedução,
        deduções, rendimento, alíquota, jazidas, dedutivo, declaração, tributária,
        renda bruta, pessoa jurídica, pessoa física
EF_154  arquitetura, arquitetos, arquiteto, arquitetônica, nanquim, renovação, mackenzie,
        contemporânea, engenharia, manifesto, drummond, moderna, urbanismo,
        faculdade de arquitetura
EF_187  avaliação, aprendizagem, objetivos, mediana, dissertação, cognitivo, critério,
        eficiência, aproveitamento, memorizar, questões, psicologia da educação, média,
        objetivas, subjetivas
F_032   autoestima, alcoholismo, adicciones, divorcio, primaria, escuela, conferencias,
        responsabilidad, pensamientos, oportunidades, experiencias, locutora, violenta,
        esencia
F_21    nitrosaminas, carcinogenicos, estomago, mamario, alimentacion, evidencia cientifica,
        alcohol, canceres, esofago, compuestos, lineamientos, recomendaciones, colon,
        tumores, panamericana

Those are the words a student of that class would need to be right. If you're testing for a different kind of recording, write your own list first, and write it from the reference, not from the transcripts, or you'll pick the words the engine already got.

The normaliser, as code

Every WER depends on decisions the formula leaves to you: case, punctuation, numbers, contractions, fillers. Two numbers made with different decisions are not comparable, and almost nobody states theirs. Here are ours. Both the reference and the transcript go through exactly this, in the clip's language:

import re, unicodedata
from num2words import num2words

FILLERS = {
    'pt': {'eh', 'éh', 'ah', 'ahn', 'ãh', 'ã', 'hum', 'hm', 'hmm', 'uhn', 'uh', 'uhm',
           'mm', 'ham', 'han', 'hã', 'hãm', 'ahm'},
    'es': {'eh', 'eeh', 'ehh', 'ah', 'ahh', 'em', 'emm', 'mm', 'mmm', 'mh', 'uh', 'um',
           'uhm', 'hm', 'hmm', 'e'},
}
# Spoken contractions that people and engines write either way: expand both sides.
SPOKEN = {
    'pt': {'pra': 'para', 'pras': 'para as', 'pro': 'para o', 'pros': 'para os',
           'tá': 'está', 'tô': 'estou', 'tava': 'estava'},
    'es': {'pa': 'para'},
}
N2W = {'pt': 'pt_BR', 'es': 'es'}

def strip_accents(s):
    return ''.join(c for c in unicodedata.normalize('NFD', s)
                   if unicodedata.category(c) != 'Mn')

def num_to_words(m, lang):
    tok = m.group(0)
    try:
        if re.fullmatch(r'\d+[.,]\d+', tok):
            a, b = re.split(r'[.,]', tok)
            sep = ' vírgula ' if lang == 'pt' else ' punto '
            return num2words(int(a), lang=N2W[lang]) + sep + num2words(int(b), lang=N2W[lang])
        return num2words(int(tok.replace('.', '')), lang=N2W[lang])
    except Exception:
        return tok

def normalise(text, lang):
    t = text.lower()
    t = t.replace('%', ' por cento ' if lang == 'pt' else ' por ciento ')
    t = re.sub(r'(\d)[ºª°]', r'\1', t)
    t = re.sub(r'\d+(?:[.,]\d+)?', lambda m: ' ' + num_to_words(m, lang) + ' ', t)
    t = t.replace('-', ' ')
    t = re.sub(r"[^\w\s]", ' ', t)
    if lang == 'es':
        t = strip_accents(t)   # the Spanish references drop accents inconsistently
    words = []
    for w in t.split():
        if w in FILLERS[lang]:
            continue
        words.extend(SPOKEN[lang].get(w, w).split())
    return ' '.join(words)

Three of those lines are arguable, and we'd rather you argue with them than not know they're there.

Fillers are removed. Whether a reference writes down every "eh" and "hum" is a transcription convention, so counting them would score the transcriber's habits as much as the engine. If you're testing for verbatim transcripts, keep them in, on both sides.

Spanish loses its accents. The CIEMPIESS references drop them inconsistently, so "está" against "esta" would be a substitution the engine didn't earn. Portuguese keeps them: we saw no reason to strip them, and "avó" and "avô" are different people.

Contractions expand. "Pra" becomes "para" on both sides. Otherwise an engine that writes what was said and a transcriber who wrote what was meant disagree every time someone says "pra".

Pin num2words at the version you started with. It spells the numbers, and a release that spells one differently moves the score without anything else changing. We use 0.5.14, and the checkpoint word counts above were made with it.

The scoring

Per clip, jiwer on the normalised strings. Keep the three error types apart, because a transcript that drops words and one that invents them are different failures, and a single rate hides which one you have:

import jiwer

def score(ref, hyp, terms):
    """ref and hyp already normalised; terms normalised the same way."""
    o = jiwer.process_words(ref, hyp)
    equal = {i for ch in o.alignments[0] if ch.type == 'equal'
             for i in range(ch.ref_start_idx, ch.ref_end_idx)}
    words, found, right = ref.split(), 0, 0
    for term in terms:
        tw = term.split()
        for i in range(len(words) - len(tw) + 1):
            if words[i:i + len(tw)] == tw:
                found += 1
                right += all(i + k in equal for k in range(len(tw)))
    return dict(wer=o.wer, cer=jiwer.cer(ref, hyp), sub=o.substitutions, dele=o.deletions,
                ins=o.insertions, ref_words=len(words), terms_found=found, terms_right=right)

The pooled figure is total errors over total reference words across the six clips, not the average of six rates. An average lets a seven-minute clip count as much as a fourteen-minute one; pooling weights each by how much was said. The per-clip figures matter more than either, for a reason that comes up below.

Running the engines

The runner is boring on purpose, and its rules are the part worth copying.

Production settings, and then without them. Every engine ran as our product would call it: the language fixed, word timestamps on, speaker labels on. One vendor's documentation warns that word timestamps can reduce accuracy, and we didn't want to assume anything about speaker labels either way. So run each engine with those features off as well: whichever of them that engine lets you turn off, one at a time if you can. Then you know what each costs on that engine, which nobody else can tell you.
Fixed languages. Brazilian Portuguese, and a Spanish variant for the Americas, on each engine, no auto-detection. A detection mistake is a real failure, but it's a different test.
Write the spend down before the call. Each call's estimated cost goes into a ledger, and the call is marked as in flight, before it is sent. A call that would take the ledger past the cap is never sent. Set the cap before you start, not after the bill.
Nothing is sent twice. A clip, engine and configuration that has a result, or that was sent and never came back cleanly, is not sent again. If the runner dies mid-call, a person checks the provider's console before anything is resent. That's slower than retrying, and it means a crash can't quietly double the bill or replace a result.
Delete the audio when you're done. An engine that reads from cloud storage needs the clips in a bucket. Give the bucket a rule that deletes objects after a few days, and delete them yourself when the run ends anyway. Ours had a three-day rule, and we deleted the clips the same day. Your own recordings would deserve the same.
Keep every raw response, errors included, and write each one down before you read it, in a file of its own. The text is what you score. The raw response, with its timestamps and speaker labels, is what you'll need the day a number looks wrong.

What running it taught us about the method

Running a test is how you find out where the test is weak. Five things we'd tell anyone using this one.

Read the clips before the total. A pooled figure over a handful of clips is mostly a figure about the hardest one, and the hardest clip need not be the same for every engine. Decide beforehand which clips look like your recordings and read those first. Then take the hardest clip out and pool again. If your conclusion changes, it rests on one recording, and you should know that before you act on it.

Plan the runs with features off from the start. We added a run with speaker labels off after the first results were in, because we hadn't planned one. Plan it. If the product uses speaker labels, compare engines with them on, and score each engine with them off too, at least on the clip with the most crosstalk, so you know what the labels cost before you compare anything else.

Rebuild the same way every time, and check that you did. Between our first run and our second, one of the two windows came back with a reference 17 words shorter after normalisation, and nobody had written down how that copy was built, so we can't say whether its audio differed too. The build above is the first run's, and rebuilding it this month matched that run exactly. That is what the checkpoints are for: run them before anything you pay for.

Speakers aren't scored at all. The corpora have no speaker reference, so the test can't say whether an engine's speaker labels are right. Counting the labels an engine returned is not scoring them: a recogniser that splits one person into three looks, by count, exactly like one that found three people.

The corpora are kinder than real audio. Segmentation already removed the long silences and the worst crosstalk, and our joins removed the pauses between utterances; TEDx keeps "good quality speech" only. The joins may also make errors of their own: some transcripts wrote a word's ending twice at a join, and that may be the join's doing rather than the engine's. The references have typos, which count as errors before any engine makes one. And six clips in two languages tell you how candidates compare on these six recordings. That's one input to a choice for lecture-shaped Portuguese and Spanish, not a verdict on either language, and nowhere near enough to rank vendors. It says nothing about English, or about a phone recording in a noisy bar.

Every number this test produces is dated. It's a measurement of the model that answered that day, and vendors update models without changing the name. Which is one more reason the test is the thing worth publishing: the score we got in October is already someone else's history.

Run it

pip install numpy soundfile pyarrow "jiwer>=4,<5" num2words==0.5.14.
Download the shards, run the build, read the references, and check the two windows against the checkpoints.
Send each WAV to each candidate with the settings you'd actually ship: the language fixed, the features you'd turn on turned on. Then again with them off.
Score with the normaliser above, unchanged, or changed and written down. Read the substitutions, deletions and insertions separately, then the term hits, then and only then the rate.
Read the errors themselves. Our word error rate calculator does the alignment in the browser and sorts the errors into numbers, likely names, negations and the rest. For Portuguese and Spanish only part of that sort works: its number-word and negation lists are English, though digits count in any language, and a likely name is guessed from a capital letter in the middle of a sentence.

If you run it and find a line in the normaliser you'd have written differently, that's a better conversation than any two percentages, and we'd like to have it.