Researchers at Hugging Face built three diagnostic tests that catch automatic speech recognition models copying a benchmark’s own errors instead of transcribing the audio in front of them. The team ran the tests on 11 widely used open source ASR systems; several of the top scorers turned out to be pattern matching the test, not the task. That gap matters because word error rate, the standard scorecard buyers use to pick a transcription vendor, can climb even as real world accuracy stays flat.
The first test, a consensus disagreement probe, exploits a known flaw in the VoxPopuli dataset (a European Parliament speech corpus): its reference transcripts contain mistakes. The researchers assembled an ensemble of models with low phoneme error rates, a measure of how closely a transcript tracks the actual sounds, and flagged clips where that ensemble unanimously disagreed with VoxPopuli’s written reference. In one clip, a speaker audibly says “Thank you, Mr. President,” but the reference transcript drops the “Thank you.” Six of the 11 models tested reproduced that omission, writing the wrong transcript even though the audio said otherwise. When the team fed the same sentence through the models again, this time spoken by a voice with no tie to any European Parliament session, most of them corrected themselves and included the missing phrase.
The second test, masked entity retrieval, silences a specific word, often a number, in the audio and checks whether a model still outputs it. In one example, the audio has “one thousand six hundred” edited out entirely, yet several models wrote the same wrong number, “1” or “2011,” found in the benchmark’s flawed reference transcript. Hugging Face reports that on LibriSpeech, the top scoring models put back a masked number about 30 to 40 percent of the time, and that share fell once the audio came from freshly collected recordings in the same source domain.
The third test, orthographic switching, checks spelling choices that sound identical, such as “Mr.” versus “Mister” or “1” versus “one.” VoxPopuli and LibriSpeech each favor a different convention across their entire test sets. Multiple models switched their spelling to match whichever benchmark was running, with some approaching 90 percent accuracy at guessing which convention a given clip expected. That is a task a model listening only to sound should perform near chance.
The common thread across all three tests is the tell: a model that reproduces a dataset’s own documented transcription errors, on audio where those errors do not apply, is not demonstrating better hearing. It is demonstrating that it can identify which benchmark it is being run against. Hugging Face’s authors found this effect weakened sharply on newly recorded audio from the same source domains, collected after the models’ training cutoffs, the strongest evidence that the behavior is memorized rather than general.
Anyone procuring a speech vendor on a single word error rate number is buying a figure this research shows can be inflated by benchmark familiarity rather than transcription skill. Ask instead whether the score holds on audio the model has never been evaluated against, and whether that held out set was split by speaker and time period rather than at random. Hugging Face has already added a “Benchmark fitting” tab to the Open ASR Leaderboard covering VoxPopuli reference errors and orthographic switching, a starting checklist for the next procurement conversation.
Hugging Face published this research on its blog on June 10, 2026, authored by Theo Lebryk, Eric Bezzam, Alice, David Ayllon, Jakub Piotr Cłapa, Jens Madsen and Panagiotis Tzirakis.
