Problems with FLEURS as a Filipino Benchmark

Part 2 of 3

This is the middle of a short series on Filipino ASR evaluation, following The Landscape of Filipino Speech Datasets. How Filipino ASR Should Be Scored closes it out.

The Benchmark Behind the Numbers

When multilingual ASR models report per-language results, the dataset they evaluate against is usually FLEURS. Whisper, MMS, SeamlessM4T, and the models on the Open ASR Leaderboard all report FLEURS scores. When companies claim their ASR system “supports Filipino,” the evidence behind that claim is a word error rate measured against FLEURS fil_ph. For Filipino, that means a 964-utterance test split of translated English Wikipedia prose read aloud by native speakers.

FLEURS (Few-shot Learning Evaluation of Universal Representations of Speech) is a 102-language speech dataset introduced by Conneau and colleagues in 2022. It provides roughly 12 hours of recorded speech per language: native speakers reading sentences aloud, three speakers per sentence, gender-balanced, quality-checked. The Filipino subset contains 3,266 utterances across its three splits: 1,884 train, 418 development, 964 test, totalling roughly 14.5 hours. The 964-utterance test split is what evaluation papers actually report against, and what this essay audits.

It is also the only freely available Filipino speech dataset with established splits, which is why it dominates evaluation despite the problems described in this essay. (For a survey of what else exists, see The Landscape of Filipino Speech Datasets.)

The problems are not incidental quality issues that affect a few utterances. They are structural properties of how the dataset was built, and they affect what the benchmark actually measures.

---

How FLEURS Was Built

The construction pipeline matters because every problem traced in this essay originates from it.

FLEURS is built on top of FLoRes (Facebook’s Low Resource translation benchmark), which consists of 2,009 sentences extracted from English Wikipedia articles. These English sentences were professionally translated into 101 other languages. FLEURS took those translations and had native speakers record them as speech. The reference transcript for each utterance is the translated sentence that the speaker was given to read.

So the pipeline is: English Wikipedia → professional translation into Filipino → native speaker reads the translation aloud → the read speech and the prompt text become the benchmark.

Three consequences follow immediately.

First, every sentence in FLEURS fil_ph is a translation. Not one was composed in Filipino. The language in the dataset is not Filipino as it is spoken, written, or thought by Filipino speakers. It is Filipino as it appears when English encyclopedic prose is converted into Filipino by a translator working from an English source.

Second, the reference transcript is the prompt, not a transcription of what the speaker actually said. If the speaker departs from the prompt (rephrasing, misreading, pronouncing an abbreviation differently than the text specifies), the reference does not reflect the departure. The “correct” transcription is defined by the translator’s text, not by the acoustic signal. This is a design choice with consequences that compound across every category in the audit below.

Third, the encyclopedic register of the source material constrains the vocabulary, syntax, and discourse patterns of the target. Wikipedia articles do not contain colloquial speech, code-switching, discourse particles, or the natural rhythms of conversation. The Filipino in FLEURS does not either.

---

Translationese: The Structural Bias

The term “translationese” refers to systematic features of translated text that distinguish it from text originally composed in the target language. The concept was formalized by Mona Baker in the 1990s and has since been extensively studied in translation studies and computational linguistics. Translated text tends to be simpler, more explicit, more standardized, and more structurally faithful to the source language than native text.

For Filipino, the translationese in FLEURS manifests in specific, observable ways.

Word order follows English rather than Filipino. English favors SVO (subject-verb-object) construction. Filipino, in its natural spoken and written forms, uses VSO and topic-comment structures extensively. A FLEURS sentence that opens with the agent, followed by the verb, followed by the object, is not ungrammatical in Filipino. But it is not how a Filipino speaker would typically frame the thought. It is how an English speaker would, and the translator preserved that framing.

Discourse particles are absent. Spoken and written Filipino makes heavy use of particles like nga, naman, pala, kasi, daw, and ba to encode emphasis, evidentiality, politeness, and interpersonal positioning. These particles have no direct English equivalents. Because the source text is English, and because translation tends to replicate what is present rather than add what is absent, the FLEURS Filipino text reads as unnaturally bare. The pragmatic texture of the language is missing.

Code-switching is absent. Natural Filipino, especially in the Manila register that constitutes the functional standard, routinely incorporates English words and phrases. A Filipino speaker discussing technology, medicine, law, or any specialized domain will code-switch without thinking about it, because the Filipino technical vocabulary for many fields is English. Wikipedia articles about these domains, when translated, produce Filipino text that either avoids code-switching (by using calques or rare Filipino equivalents) or includes it inconsistently based on the translator’s preferences. Neither outcome reflects how a Filipino speaker would discuss the same topic aloud.

This is not a marginal concern. Montalan et al. (ACL 2025, “Batayan: A Filipino NLP benchmark for evaluating Large Language Models”) explicitly identify “the pervasive translationese bias in existing Filipino corpora” as a problem their benchmark was designed to address, using native-speaker-driven annotation rather than translation-based corpus construction.

The consequence for ASR evaluation is direct. A model trained on natural Filipino speech will produce output that reflects natural Filipino patterns: VSO constructions, discourse particles, code-switching. When evaluated against FLEURS references that lack these features, the model’s natural output is penalized. Conversely, a model that has learned to reproduce the translationese patterns of the training data will score well on FLEURS, not because it understands Filipino better, but because its errors align with the benchmark’s biases.

---

The Audit: Ten Categories of Reference Quality Issues

To move beyond structural argument to empirical evidence, I conducted a systematic audit of 100 randomly selected utterances from the FLEURS fil_ph dataset. The sample size was chosen for statistical defensibility: at a 95% confidence level with a 10% margin of error, the standard heuristic calls for 96 samples. The 964-utterance test split, the portion under evaluation here, comfortably supports a sample of 100. The utterances were selected using a prompt randomizer tool to eliminate selection bias.

Each utterance was examined by comparing the reference transcript against the actual speech. The governing filter: does this property of the reference text cause the scorer to misrepresent the model’s actual recognition performance? Specifically, does it affect the acoustic signal, or does it affect what a language model would predict? If the answer to both is no, it does not belong in the audit.

Ten categories of issues emerged. The manual audit findings were then validated and quantified through a cross-system alignment analysis of ten ASR systems spanning five distinct architectures (Whisper, Azure Speech, MMS, SeamlessM4T, Kaldi/Vosk) plus FlipVox-trained zipformer models, run against the full 964-utterance test set. When multiple independently trained models agree on an output that differs from the reference, the reference is on trial, not the models.

Every consensus figure quoted below (the n/10 counts and the substitution totals) comes from that ten-system run, conducted in August 2026. The evaluated pool has since grown past thirty systems, adding cloud APIs, OWSM, Dolphin, OmniASR, Qwen3-ASR and more zipformer variants. Those newer systems are not folded into the counts here, and the counts should be read as belonging to the run that produced them. Their addition has not disturbed the conclusion; if anything a wider and more architecturally diverse pool agreeing on the same substitutions strengthens it.

1. Mixed code-switching orthography

When English words appear in the Filipino text, the transcription is inconsistent about whether to spell them in English orthography or Filipinize them. The same English loanword might appear as its English spelling in one utterance and a phonetic Filipino spelling in another, with no pattern governing the choice.

The cross-system evidence is striking. All ten models unanimously output “cake” where the reference has “keyk” (28 total substitutions, 10/10 consensus). Nine out of ten produce “screen” for “iskrin” (44 total), and the same pattern repeats across every English-origin term the reference chose to respell. The models are not wrong. They are producing a legitimate orthographic rendering of the same word. The reference chose one convention; the models chose another; neither is incorrect.

This is not a flaw in any individual transcriber’s judgment. It reflects the absence of a community standard for loanword orthography in Filipino, which the benchmark inherits without acknowledging. The Komisyon sa Wikang Filipino’s Ortograpiyang Pambansa (2014 edition) attempts to address this: Section 4.7 encourages experimentation in respelling borrowed English words, while Section 4.6 preserves foreign proper nouns in their original spelling. But the guide does not resolve the boundary cases that affect ASR scoring. Is “Catalan” a proper noun (keep as-is, per Section 4.6) or a common term (respell as “Katalan”)? Seven out of ten systems output “Katalan.” The Ortograpiyang Pambansa does not say who is right.

2. Translationese affecting naturalness and coarticulation

Beyond the structural translationese described above, specific sentences exhibit phrasing that no native speaker would produce unprompted. When speakers read these sentences aloud, their prosody and coarticulation patterns reflect the unnaturalness: hesitations at syntactic boundaries that would not exist in native Filipino, unnatural stress patterns on calqued phrases, and reading rhythms that suggest the speaker is parsing the sentence in real time rather than producing memorized or familiar constructions.

This matters for ASR because coarticulation (the way sounds blend into each other in natural speech) is one of the acoustic cues that models rely on. When coarticulation patterns are disrupted by unnatural syntax, the acoustic signal itself becomes less representative of the language the model is meant to serve.

3. Speaker preferred pronunciation

Speakers sometimes pronounce words differently than the reference text specifies, not because they are making errors but because they are applying standard Filipino pronunciation conventions that the written reference does not capture. The most common case is abbreviation expansion: a reference that contains “PM” might be read aloud as “prime minister” or as the Filipino “hapon” (afternoon), depending on the speaker’s interpretation of the context. The reference records “PM.” The speaker said something else entirely. Under standard WER scoring, this is a substitution error. Under any reasonable evaluation standard, it is a correct reading.

A subtler variant surfaced during the audit: native phonological processes applied to common words. The word “selula” (cell) appeared twice in one utterance; the speaker pronounced it with a neutral /lu/ the first time and with a glide-inserted /lyu/ (as if “selyula”) the second time. This glide insertion is a productive Filipino phonological process, not an error. Nine out of ten ASR systems produced “selyula” for this word, faithfully transcribing what was spoken. All nine are penalized because the reference accommodates only “selula.”

4. Proper casing conventions

The reference transcripts use casing conventions that are irrelevant to speech recognition but become relevant when scoring tools are case-sensitive. Some evaluation setups lowercase everything before scoring; others do not.

The cross-system evidence reveals the scale of this problem. The substitution Ang → ang is the single largest source of measured error across the entire dataset: 2,668 total substitutions, unanimous across all ten systems. Every system correctly recognized the word “ang.” The scorer penalizes them because the reference capitalizes sentence-initial words and the models do not. The same unanimous pattern repeats across dozens of other sentence-initial capitalized words. If the scoring pipeline is case-sensitive, casing alone inflates WER by thousands of tokens without measuring recognition.

5. Loanword diacritic handling

Spanish-derived Filipino words sometimes retain diacritics (accents, tildes) and sometimes do not. The reference transcripts are inconsistent about this, and the inconsistency creates scoring artifacts: a model that outputs the unaccented form when the reference has the accented form (or vice versa) is penalized for a substitution that reflects typographic convention, not acoustic misrecognition.

6. Orthographic preference variants

This is the category with the most direct impact on WER scores. Filipino has numerous word pairs where two spellings represent the same word, and neither is incorrect. The audit and cross-system analysis identified twelve high-frequency pairs:

Each of these pairs, when the model produces one form and the reference contains the other, registers as a substitution error. The top five pairs alone account for over 600 substitutions across ten systems. In aggregate, these orthographic disagreements account for a substantial portion of what WER reports as recognition failure. (For a detailed treatment of how scoring should handle these variants, see How Filipino ASR Should Be Scored.)

A separate cluster deserves mention. The function particles ng, na, and nang are the connective tissue of Filipino syntax. They appear in nearly every sentence, and their acoustic realizations overlap heavily in connected speech. The substitution nang → ng alone produces 433 errors across nine of ten systems. Unlike the pairs above, nang and ng are grammatically distinct words (one is an adverb marker, the other a genitive marker). But they are often acoustically indistinguishable. A scorer that penalizes this substitution is measuring orthographic-grammatical knowledge, not speech recognition.

7. Numeral rendering mismatches

Numbers in the reference text are written as digits. Speakers read them as words. When the reference contains “50 kilometro” and the speaker says “limampung kilometro,” minimum edit distance charges a single substitution for a unit that was correctly recognized. Where a digit expands into two words (“100” against isang daan), the cost is a substitution plus an insertion. The penalty is real but smaller than it first appears; what makes it serious is how often it fires, and that no scoring rule can remove it (see How Filipino ASR Should Be Scored).

The cross-system data confirms this at scale: 10 → sampung is unanimous across all ten systems (24 total substitutions). But the numeral problem goes deeper than digit-to-word expansion. Filipino does not use a single number system. Sports scores are rendered in English (“six-all,” never “anim-anim”). Years are historically read in Spanish (“mil novecientos sesenta”). Prices sometimes in English, sometimes in Filipino. The translation pipeline flattens all of this into Filipino number words, producing text that is grammatically valid but pragmatically foreign. In one utterance about a Del Potro tennis match, the score “6-6” was read as “anim anim.” No Filipino watching tennis would say it this way.

8. Parenthetical content ignored during reading

Some FLEURS sentences contain parenthetical text that speakers skip when reading aloud. The reference includes the parenthetical. The speech does not. Every word in the skipped parenthetical registers as a deletion, penalizing the model for faithfully transcribing what was actually spoken.

The problem resists simple normalization. In one utterance, the speaker skipped one parenthetical (“palagi”) but read another (“kapag maaliwalas ang langit, mas madalang”) in the same sentence, making a real-time judgment about which was essential to meaning. A blanket rule to strip parentheticals would fix some cases and break others.

9. Written-form typographic artifacts

Reference text carries typographic conventions from the Wikipedia source (quotation marks, hyphens, special punctuation) that have no acoustic realization.

The cross-system data reveals three sub-classes of this problem. First, quotation marks: all ten systems unanimously strip them, producing substitutions like “mga → mga (57 total), and repeating across every quoted or comma-adjacent word in the set, because the scorer sees a word with an attached quotation mark as a different token from the same word without one. Second, comma handling: nito, → nito produces 147 substitutions at 10/10 consensus, with the pattern repeating across dozens of comma-bearing words. Third, hyphenated compound splitting: Filipino uses hyphens in compound words (panukalang-batas, carbon-dioxide, hukbong-pandagat), but the hyphen carries no acoustic signal. All ten systems split these into their component parts. panukalang-batas → panukalang alone accounts for 55 substitutions at 10/10 consensus.

Whether these register as errors depends on the scoring pipeline’s punctuation handling. But across the ten systems tested, they do register, and they inflate WER without measuring anything about recognition.

10. Speaker misreads

Speakers occasionally depart from the prompt: swapping a prefix (maka- for naka-, or vice versa), reading a year as individual digits, simplifying a plural form, or producing disfluencies. These are genuine speaker departures, and a transcription-based reference would record them accurately. Because FLEURS uses the prompt as the reference, the speaker’s actual production is invisible. The model is evaluated against text that does not describe the audio it processed.

The cross-system evidence confirms the most telling pattern. The reference labels are inconsistent about adjective pluralization: in one utterance the reference has “maliliit” (plural) but the speaker said “maliit” (singular); in another the reference has “maliit” but the speaker said “maliliit.” Both directions appear at 10/10 consensus (40 and 37 total substitutions respectively). Every system participates in both. Speakers instinctively normalize toward whatever form sounds natural in context, regardless of what the prompt says, and the reference itself cannot decide which form is correct.

One case is categorically different from the others. In one utterance, the reference has “at halos ay lahat alam” but the speaker read “at halos lahat ay alam,” correcting the position of the “ay” marker. The reference is wrong both as a transcription of the audio and as Filipino. A model that correctly transcribes the speech is penalized for producing correct output.

---

What the Reference Does Not Declare

Beneath the ten categories lies a question that FLEURS never answers: what does the reference transcript represent?

There are three possible answers, and each produces different WER numbers measuring different things. The reference could represent what the speaker was supposed to say (the prompt). It could represent what the speaker actually said (a verbatim transcript). Or it could represent what the speaker intended to communicate (a canonical, cleaned transcript). FLEURS chose the first option, which is the cheapest to produce (no transcription step at all) and the most problematic to evaluate against.

This choice conflates translation quality with recognition quality. Categories 1 through 9 in the audit are all properties of the prompt, not properties of the speech. Every one of them can inflate or deflate WER without any change in model performance. And when the speaker departs from the prompt (Category 10), the reference cannot represent what actually happened, because it was never designed to.

The problem compounds when the reference must serve multiple evaluation objectives. Faithfully representing a speaker disfluency helps acoustic model evaluation but corrupts language model probability estimates and downstream task utility. The conflict is not orthographic; it is architectural. Hub-4, Fisher, and Switchboard solved this decades ago with annotation tiers: a verbatim transcript with disfluency markup alongside a canonical clean transcript. FLEURS has no such mechanism. There is no established Filipino disfluency annotation convention for it to draw on even if it tried.

---

What the Audit Tells Us

The ten categories are not independent. A single utterance can exhibit translationese (category 2), a code-switching orthography inconsistency (category 1), an orthographic preference variant (category 6), and a typographic artifact (category 9) simultaneously. The effect on WER is cumulative: a model that correctly transcribes the speech in a natural, code-switched, orthographically reasonable way will accumulate errors across multiple categories, not because it failed at recognition but because its output does not match the reference’s arbitrary conventions.

The cross-system alignment analysis puts a floor under this claim. When all ten independently trained systems, spanning five distinct architectures plus FlipVox-trained models, unanimously produce the same substitution, the measured “error” is a property of the reference, not of the models. Casing alone (Ang/ang and its cousins) contributes thousands of substitutions. Orthographic variants contribute hundreds more. Typographic artifacts (quotation marks, commas, hyphens) contribute hundreds more still. Before any genuine recognition error is counted, the WER figure is already carrying substantial noise from conventions the reference chose and the models did not.

The findings are consistent with a broader pattern documented across languages. Lau et al. (2025, “Data Quality Issues in Multilingual Speech Datasets,” arXiv 2506.17525) conducted a quality audit of Common Voice 17.0, FLEURS, and VoxPopuli, finding that quality problems in these datasets create what they call an “illusion of success” in low-resource languages. Their sharpest parallel to the Filipino case involves Norwegian: FLEURS does not keep Bokmål and Nynorsk cleanly separated (the Bokmål subset is approximately 8.8% Nynorsk), and a model trained on Bokmål showed a substitution rate inflation of nearly 25 percentage points from this mismatch alone. When two orthographic conventions coexist in the reference and the model follows one consistently, WER punishes consistency.

---

What This Means in Practice

The practical consequence is that FLEURS fil_ph WER figures should not be taken at face value. They measure something, but that something is not “how well does this model transcribe Filipino speech.” It is closer to “how closely does this model’s output match the orthographic conventions of a single translator working from English Wikipedia source text.”

A low WER on FLEURS does not guarantee good performance on conversational Filipino, code-switched Filipino, or any register of Filipino other than the one the dataset contains. A high WER on FLEURS does not necessarily indicate poor recognition quality; it may indicate that the model produces natural Filipino output that the reference does not expect.

The FlipVox ASR Leaderboard on flipvox.ph reports FLEURS scores because, despite its problems, FLEURS remains the only freely available Filipino evaluation set with established splits. But it reports them under multiple normalization conditions, with explicit documentation of how each score was computed. Broadening beyond read speech has since begun: a broadcast-speech set (OpenASR21 Tagalog, MATERIAL) has now been evaluated across the same pool of systems. The Landscape of Filipino Speech Datasets discusses what that comparison shows so far and why it isn’t clean evidence yet. Conversational telephone speech remains a gap. The goal is not to hide FLEURS’s limitations but to make them visible by showing how the same models perform under different conditions and against different references.

---

What Would Be Better

The Filipino speech research community needs an evaluation dataset that was not translated from English. The sentences should originate in Filipino: drawn from news broadcasts, podcasts, conversations, lectures, public discourse, and the full range of registers that Filipino speakers actually use. The recordings should capture natural speech, including code-switching, discourse particles, and the full prosodic repertoire of the language. The reference transcripts should be derived from the audio, not from a prompt text that the speaker was asked to read.

Any such dataset must declare a reference specification: what the reference transcript represents, what annotation conventions govern phenomena the reference cannot express in plain text (disfluencies, partial words, restarts), and how evaluation targets are separated when a single reference string cannot serve all of them. The Ortograpiyang Pambansa’s unresolved boundary cases mean that the transcription manual for a Filipino speech dataset becomes a policy document, not just a style guide. The choices it makes (Catalan or Katalan, equator or ekwador, computer or kompyuter) directly determine what the scorer treats as correct.

Building such a dataset is not trivial. It requires recording infrastructure, transcription guidelines, quality control, and sustained effort. But the alternative, continuing to evaluate Filipino ASR against a benchmark that structurally misrepresents the language, is not sustainable either.

Until that dataset exists, anyone interpreting Filipino ASR scores should ask three questions: What was the evaluation set? How were the references constructed? And what normalization was applied before scoring? The answers to those questions will tell you more about the number than the number itself.