The Benchmark Behind the Numbers
When multilingual ASR models report per-language results, the dataset they evaluate against is usually FLEURS. Whisper, MMS, SeamlessM4T, and the models on the Open ASR Leaderboard all report FLEURS scores. When companies claim their ASR system “supports Filipino,” the evidence behind that claim is a word error rate measured against FLEURS fil_ph. For Filipino, that means 964 utterances of translated English Wikipedia prose read aloud by native speakers.
FLEURS (Few-shot Learning Evaluation of Universal Representations of Speech) is a 102-language speech dataset introduced by Conneau and colleagues in 2022. It provides roughly 12 hours of recorded speech per language: native speakers reading sentences aloud, three speakers per sentence, gender-balanced, quality-checked. The Filipino subset contains 964 utterances across train, development, and test splits.
It is also the only freely available Filipino speech dataset with established splits, which is why it dominates evaluation despite the problems described in this essay. (For a survey of what else exists, see The Landscape of Filipino Speech Datasets.)
The problems are not incidental quality issues that affect a few utterances. They are structural properties of how the dataset was built, and they affect what the benchmark actually measures.
How FLEURS Was Built
The construction pipeline matters because every problem traced in this essay originates from it.
FLEURS is built on top of FLoRes (Facebook’s Low Resource translation benchmark), which consists of 2,009 sentences extracted from English Wikipedia articles. These English sentences were professionally translated into 101 other languages. FLEURS took those translations and had native speakers record them as speech. The reference transcript for each utterance is the translated sentence that the speaker was given to read.
So the pipeline is: English Wikipedia → professional translation into Filipino → native speaker reads the translation aloud → the read speech and the prompt text become the benchmark.
Three consequences follow immediately.
First, every sentence in FLEURS fil_ph is a translation. Not one was composed in Filipino. The language in the dataset is not Filipino as it is spoken, written, or thought by Filipino speakers. It is Filipino as it appears when English encyclopedic prose is converted into Filipino by a translator working from an English source.
Second, the reference transcript is the prompt, not a transcription of what the speaker actually said. If the speaker departs from the prompt (rephrasing, misreading, pronouncing an abbreviation differently than the text specifies), the reference does not reflect the departure. The “correct” transcription is defined by the translator’s text, not by the acoustic signal.
Third, the encyclopedic register of the source material constrains the vocabulary, syntax, and discourse patterns of the target. Wikipedia articles do not contain colloquial speech, code-switching, discourse particles, or the natural rhythms of conversation. The Filipino in FLEURS does not either.
Translationese: The Structural Bias
The term “translationese” refers to systematic features of translated text that distinguish it from text originally composed in the target language. The concept was formalized by Mona Baker in the 1990s and has since been extensively studied in translation studies and computational linguistics. Translated text tends to be simpler, more explicit, more standardized, and more structurally faithful to the source language than native text.
For Filipino, the translationese in FLEURS manifests in specific, observable ways.
Word order follows English rather than Filipino. English favors SVO (subject-verb-object) construction. Filipino, in its natural spoken and written forms, uses VSO and topic-comment structures extensively. A FLEURS sentence that opens with the agent, followed by the verb, followed by the object, is not ungrammatical in Filipino. But it is not how a Filipino speaker would typically frame the thought. It is how an English speaker would, and the translator preserved that framing.
Discourse particles are absent. Spoken and written Filipino makes heavy use of particles like nga, naman, pala, kasi, daw, and ba to encode emphasis, evidentiality, politeness, and interpersonal positioning. These particles have no direct English equivalents. Because the source text is English, and because translation tends to replicate what is present rather than add what is absent, the FLEURS Filipino text reads as unnaturally bare. The pragmatic texture of the language is missing.
Code-switching is absent. Natural Filipino, especially in the Manila register that constitutes the functional standard, routinely incorporates English words and phrases. A Filipino speaker discussing technology, medicine, law, or any specialized domain will code-switch without thinking about it, because the Filipino technical vocabulary for many fields is English. Wikipedia articles about these domains, when translated, produce Filipino text that either avoids code-switching (by using calques or rare Filipino equivalents) or includes it inconsistently based on the translator’s preferences. Neither outcome reflects how a Filipino speaker would discuss the same topic aloud.
This is not a marginal concern. Montalan et al. (ACL 2025, “Batayan: A Filipino NLP benchmark for evaluating Large Language Models”) explicitly identify “the pervasive translationese bias in existing Filipino corpora” as a problem their benchmark was designed to address, using native-speaker-driven annotation rather than translation-based corpus construction.
The consequence for ASR evaluation is direct. A model trained on natural Filipino speech will produce output that reflects natural Filipino patterns: VSO constructions, discourse particles, code-switching. When evaluated against FLEURS references that lack these features, the model’s natural output is penalized. Conversely, a model that has learned to reproduce the translationese patterns of the training data will score well on FLEURS, not because it understands Filipino better, but because its errors align with the benchmark’s biases.
The Audit: Nine Categories of Reference Quality Issues
To move beyond structural argument to empirical evidence, I conducted a systematic audit of 100 randomly selected utterances from the FLEURS fil_ph dataset. The sample size was chosen for statistical defensibility: at a 95% confidence level with a 10% margin of error, 96 samples are needed from a population of 964. The utterances were selected using a prompt randomizer tool to eliminate selection bias.
Each utterance was examined by comparing the reference transcript (the prompt text) against the actual speech, with attention to cases where the reference misrepresents what was spoken, where the orthographic conventions are inconsistent, and where the evaluation framework would produce misleading scores.
Nine categories of issues emerged.
1. Mixed code-switching orthography
When English words appear in the Filipino text, the transcription is inconsistent about whether to spell them in English orthography or Filipinize them. The same English loanword might appear as its English spelling in one utterance and a phonetic Filipino spelling in another, with no pattern governing the choice. This is not a flaw in the individual transcriber’s judgment; it reflects the absence of a community standard for loanword orthography in Filipino, which the benchmark inherits without acknowledging.
2. Translationese affecting naturalness and coarticulation
Beyond the structural translationese described above, specific sentences exhibit phrasing that no native speaker would produce unprompted. When speakers read these sentences aloud, their prosody and coarticulation patterns reflect the unnaturalness: hesitations at syntactic boundaries that would not exist in native Filipino, unnatural stress patterns on calqued phrases, and reading rhythms that suggest the speaker is parsing the sentence in real time rather than producing memorized or familiar constructions.
This matters for ASR because coarticulation (the way sounds blend into each other in natural speech) is one of the acoustic cues that models rely on. When coarticulation patterns are disrupted by unnatural syntax, the acoustic signal itself becomes less representative of the language the model is meant to serve.
3. Speaker preferred pronunciation
Speakers sometimes pronounce words differently than the reference text specifies, not because they are making errors but because they are applying standard Filipino pronunciation conventions that the written reference does not capture. The most common case is abbreviation expansion: a reference that contains “PM” might be read aloud as “prime minister” or as the Filipino “hapon” (afternoon), depending on the speaker’s interpretation of the context. The reference records “PM.” The speaker said something else entirely. Under standard WER scoring, this is a substitution error. Under any reasonable evaluation standard, it is a correct reading.
4. Proper casing conventions
The reference transcripts use casing conventions (capitalizing proper nouns, sentence-initial capitals) that are irrelevant to speech recognition but become relevant when scoring tools are case-sensitive or when casing interacts with normalization rules. Some evaluation setups lowercase everything before scoring; others do not. The inconsistency across the field means that the same model output can produce different WER figures depending on an implementation detail that has nothing to do with recognition quality.
5. Loanword diacritic handling
Spanish-derived Filipino words sometimes retain diacritics (accents, tildes) and sometimes do not. The reference transcripts are inconsistent about this, and the inconsistency creates scoring artifacts: a model that outputs the unaccented form when the reference has the accented form (or vice versa) is penalized for a substitution that reflects typographic convention, not acoustic misrecognition.
6. Orthographic preference variants
This is the category with the most direct impact on WER scores. Filipino has numerous word pairs where two spellings represent the same word, and neither is incorrect. The audit identified several high-frequency examples:
| Variant A | Variant B | Nature of variation |
|---|---|---|
| kanyang | kaniyang | Glide elision |
| nyang | niyang | Glide elision |
| ngunit | nguni’t | Glottal stop marking |
| bawat | bawa’t | Glottal stop marking |
| gayundin | gayun din | Segmentation |
| ng | nang | Homophonous function words |
Each of these pairs, when the model produces one form and the reference contains the other, registers as a substitution error. In aggregate, these orthographic disagreements account for a substantial portion of what WER reports as recognition failure. (For a detailed treatment of how scoring should handle these variants, see How Filipino ASR Should Be Scored.)
7. Numeral rendering mismatches
Numbers in the reference text are sometimes written as digits and sometimes spelled out. When the reference contains “100” and the model outputs “isang daan,” the scorer registers one deletion and two insertions (or some combination of substitutions and insertions, depending on alignment), inflating the error count by three words for a single semantic unit that was correctly recognized.
8. Parenthetical content ignored during reading
Some FLEURS sentences contain parenthetical text (material enclosed in parentheses or set off by punctuation) that speakers skip when reading aloud. The reference includes the parenthetical. The speech does not. Every word in the skipped parenthetical registers as a deletion, penalizing the model for faithfully transcribing what was actually spoken rather than what the prompt text contained.
This is a direct consequence of FLEURS using the prompt as the reference rather than a transcription of the speech. A dataset that transcribed from the audio would not have this problem.
9. Speaker misreads
Speakers occasionally misread words: swapping a prefix (m for n, or vice versa), reading a year as individual digits rather than as a number (“dalawang libo at isa” vs. “two-zero-zero-one”), or producing a different but phonetically similar word. These are genuine speaker errors, and a transcription-based reference would record them accurately. Because FLEURS uses the prompt as the reference, the speaker’s actual production is invisible. The model is evaluated against text that does not describe the audio it processed.
What the Audit Tells Us
The nine categories are not independent. A single utterance can exhibit translationese (category 2), a loanword diacritic inconsistency (category 5), and an orthographic preference variant (category 6) simultaneously. The effect on WER is cumulative: a model that correctly transcribes the speech in a natural, code-switched, orthographically reasonable way will accumulate errors across multiple categories, not because it failed at recognition but because its output does not match the reference’s arbitrary conventions.
The audit findings are consistent with a broader pattern documented across languages. Lau et al. (2025, “Data Quality Issues in Multilingual Speech Datasets,” arXiv 2506.17525) conducted a quality audit of Common Voice 17.0, FLEURS, and VoxPopuli, finding that quality problems in these datasets create what they call an “illusion of success” in low-resource languages. Their sharpest parallel to the Filipino case involves Norwegian: FLEURS does not keep Bokmål and Nynorsk cleanly separated (the Bokmål subset is approximately 8.8% Nynorsk), and a model trained on Bokmål showed a substitution rate inflation of nearly 25 percentage points from this mismatch alone. When two orthographic conventions coexist in the reference and the model follows one consistently, WER punishes consistency.
Szymański et al. (EMNLP 2020, “WER we are and WER we think we are”) made the broader case that benchmark WER figures for ASR consistently understate real-world error rates, and that the field needs evaluation sets built from spontaneous, multi-domain speech with high-quality annotations. Six years later, Filipino still lacks such a dataset.
What This Means in Practice
The practical consequence is that FLEURS fil_ph WER figures should not be taken at face value. They measure something, but that something is not “how well does this model transcribe Filipino speech.” It is closer to “how closely does this model’s output match the orthographic conventions of a single translator working from English Wikipedia source text.”
A low WER on FLEURS does not guarantee good performance on conversational Filipino, code-switched Filipino, or any register of Filipino other than the one the dataset contains. A high WER on FLEURS does not necessarily indicate poor recognition quality; it may indicate that the model produces natural Filipino output that the reference does not expect.
The FlipVox ASR Leaderboard on flipvox.ph reports FLEURS scores because, despite its problems, FLEURS remains the only freely available Filipino evaluation set with established splits. But it reports them alongside scores on other test sets (including conversational telephone speech from BABEL and broadcast speech from MATERIAL), under multiple normalization conditions, with explicit documentation of how each score was computed. The goal is not to hide FLEURS’s limitations but to make them visible by showing how the same models perform under different conditions and against different references.
What Would Be Better
The Filipino speech research community needs an evaluation dataset that was not translated from English. The sentences should originate in Filipino: drawn from news broadcasts, podcasts, conversations, lectures, public discourse, and the full range of registers that Filipino speakers actually use. The recordings should capture natural speech, including code-switching, discourse particles, and the full prosodic repertoire of the language. The reference transcripts should be derived from the audio, not from a prompt text that the speaker was asked to read.
Building such a dataset is not trivial. It requires recording infrastructure, transcription guidelines, quality control, and sustained effort. But the alternative, continuing to evaluate Filipino ASR against a benchmark that structurally misrepresents the language, is not sustainable either.
Until that dataset exists, anyone interpreting Filipino ASR scores should ask three questions: What was the evaluation set? How were the references constructed? And what normalization was applied before scoring? The answers to those questions will tell you more about the number than the number itself.
This essay is part of a series on Filipino ASR evaluation. See also: The Landscape of Filipino Speech Datasets and How Filipino ASR Should Be Scored. The audit described here informs a forthcoming arXiv paper on Filipino ASR evaluation methodology.