How Filipino ASR Should Be Scored

Part 3 of 3

This closes a short series on Filipino ASR evaluation, following The Landscape of Filipino Speech Datasets and Problems with FLEURS as a Filipino Benchmark. Those two survey the data and diagnose its flagship benchmark; this one sets out how to score anyway.

The Problem in One Sentence

When you evaluate a Filipino ASR system using standard tools and default settings, a significant portion of what the scorer reports as errors are not recognition failures. They are disagreements about how to spell the same word.

This essay explains how that happens, why the field’s standard evaluation tools lack the machinery to prevent it, and what a proper Filipino scoring protocol would look like. That protocol runs out partway through: past a certain point, the fix stops being about how you score and becomes about how the reference was written in the first place.

What WER Assumes

Word Error Rate, the dominant metric in speech recognition evaluation since the 1990s, is computed by aligning a reference transcript against a hypothesis (the system’s output) and counting substitutions, deletions, and insertions. The formula is simple: the sum of those three error types, divided by the number of words in the reference.

The simplicity is deceptive. WER assumes that the reference and the hypothesis share the same orthographic conventions. If the reference spells a word one way and the system produces a legitimate alternative spelling, WER counts it as a substitution. If the reference includes a hesitation marker and the system omits it, WER counts a deletion. If the reference writes a number as digits and the system spells it out, every word in the spelled-out form becomes an insertion or substitution. None of these are recognition failures. All of them inflate the error rate.

For English, this problem is manageable because English orthography is relatively standardized. For Filipino, it is not.

What the Field Built and Then Forgot

The mismatch between what a model produces and how a reference represents the same speech is not a new problem. It was identified and solved, procedurally, in the 1990s.

The NIST Speech Technology Evaluation program, which ran the Hub-4 broadcast news and Switchboard evaluations that shaped a generation of ASR research, developed a scoring toolkit called SCTK (the NIST Scoring Toolkit). Its core program, sclite, implemented Levenshtein alignment, the same dynamic programming algorithm behind WER. But sclite did not stop at alignment. It supported a rich set of mechanisms for handling legitimate variation between reference and hypothesis.

GLM files (Global Language Mapping) allowed evaluators to define equivalence classes before scoring. Contractions could be mapped to their expanded forms, spelling variants could be declared equivalent, and the resulting alternations were embedded directly into the reference text. When sclite encountered a word that had been mapped to multiple acceptable forms, it chose the alignment that minimized the error count. The scorer knew, because a human had told it, that “gonna” and “going to” were the same utterance.

STM format (Segment Time Mark) supported inline alternations using a {text / alt} syntax, a null-word symbol whose insertion was not counted as an error, and words marked optional with parentheses. Optional-word scoring treated marked reference or hypothesis words as optionally deletable, so that a hesitation filler in the reference would not penalize a system that (correctly) omitted it.

CTM format (Conversation Time Mark) carried ALT blocks representing alternative decodings, typically created by GLM filtering.

Disfluency protocols were explicit. In the Hub-4 evaluations, pause fillers were removed from references via a filter script before scoring. Overlapping-speech regions were tagged and excluded from scoring entirely. These were not informal conventions. They were published evaluation plans that participating labs agreed to follow.

The entire architecture reflected a basic insight: the question “did the system get it right?” cannot be answered by string matching alone. It requires a definition of what counts as equivalent, and that definition must be encoded somewhere the scorer can reach it.

This infrastructure was not perfect. sclite’s C source has known bugs, including one where doubly-nested alternatives after GLM filtering fail to expand correctly. Researchers as recently as 2022 were patching the original C code to work around these issues. No widely used Python reimplementation of sclite exists.

But the conceptual framework was sound, and for twenty years it was standard practice.

What Replaced It

The end-to-end revolution in ASR, beginning roughly with Deep Speech in 2014 and accelerating through wav2vec 2.0, HuBERT, and Whisper, changed not just how models were built but how they were evaluated. The new models output UTF-8 text directly. The old Viterbi-decoded lattices and word graphs are gone. And with them, the ecosystem of scoring tools that operated on those structured representations largely disappeared from common practice.

The replacement, in most published work and open-source pipelines, is a single function call. In Python, the dominant package is jiwer, which computes WER in one line:

wer = jiwer.wer(reference, hypothesis)

jiwer performs basic text normalization: lowercasing, punctuation removal, whitespace collapsing. It does not support alternations. It does not support optional words. It does not support GLM-style equivalence mappings. It does not have a concept of language-specific normalization rules. It computes Levenshtein distance on two strings and returns a number.

This is not a criticism of jiwer as software. It does what it was designed to do. The problem is that it is now the standard evaluation tool for languages whose orthography demands exactly the scoring flexibility that jiwer does not provide.

Whisper made this worse by establishing a normalization convention that the rest of the field adopted. Whisper ships with two text normalizers: an English normalizer that performs rich, linguistically informed transformations (expanding contractions, normalizing numbers, handling currencies and abbreviations), and a “basic” normalizer for all other languages that strips punctuation, lowercases, and removes Unicode characters in the mark class.

That last operation is catastrophic for languages written in Indic scripts. Manohar and Pillai (EMNLP 2024, “What is lost in Normalization?”) demonstrated that Whisper’s normalization removes vowel signs (matras) from Indic scripts, producing artificial WER reductions between 10 and 34 percentage points. The same normalization routine has been adopted by MMS, SeamlessM4T, and AssemblyAI. A model that destroys vowel signs and then reports low WER under a normalizer that also destroys vowel signs has not demonstrated accurate transcription. It has demonstrated consistent destruction.

Filipino, written in Latin script, does not suffer from the vowel-sign problem. But it suffers from the same underlying asymmetry: the English normalizer was built by people who understand English; the multilingual normalizer was built by people who needed something that would not crash on non-English text. Nobody wrote the normalizer that knows Filipino.

What Filipino Needs

Filipino’s orthographic variation falls into several classes, each of which can be a rule or a set of rules in a GLM file. (One apparent class, numeral rendering, turns out not to belong here at all; it is treated separately below.) The examples below are drawn from a systematic audit of the FLEURS fil_ph evaluation set (documented separately in Problems with FLEURS as a Filipino Benchmark).

Clitic and affix boundary variation. Filipino has enclitic particles and ligatures that can be written as part of the preceding word or as separate tokens. The most common case is the linker na/ng, which appears as -ng suffixed to vowel-final words (e.g., malaking from malaki + na) or as a separate word na after consonant-final words. But the boundary is not always observed consistently, and compounds involving din/rin, daw/raw, and lang/lamang can be written as one word or two. A GLM rule would declare these forms equivalent.

The FLEURS audit tabulates twelve such pairs in full, most with cross-system consensus counts (see Problems with FLEURS as a Filipino Benchmark). Two more are worth calling out here on their own terms:

The ng/nang pair deserves particular attention. These two words are homophonous in spoken Filipino: both are pronounced /naŋ/. The distinction is purely orthographic (and, for careful writers, grammatical: ng marks a nominal complement, nang marks an adverbial or temporal modifier). A model that outputs ng where the reference has nang, or vice versa, has not made an acoustic error. It has made an orthographic choice that even native speakers frequently disagree on. Any scoring treatment of this pair is a policy decision, not a ground truth.

Code-switched loanword spelling. Filipino speech routinely incorporates English words, Spanish-derived vocabulary, and terms from other Philippine languages. The orthographic treatment of these words is unstandardized. “Computer” might appear as kompyuter, computer, or kumpyuter depending on the transcriber’s preference and the register of the text. Spanish-derived words like presidente coexist with the Filipinized pangulo. A GLM rule cannot enumerate every possible loanword variant, but it can handle the high-frequency cases that dominate error counts.

Disfluency handling. Conversational Filipino, like any natural speech, includes hesitation fillers (ah, eh, ano, kasi), false starts, and repairs. Some of these (particularly ano and kasi) are also content words in other contexts, making automatic removal risky. A scoring protocol needs explicit rules for which fillers to exclude and under what conditions, not a blanket strip-all-hesitations approach.

Numerals Are a Different Problem

Numeral rendering looks like it belongs on the list above, and it does not. The distinction matters enough to state plainly, because getting it wrong sends the work in the wrong direction.

Numbers can be written as digits or spelled out, and the spelled-out forms in Filipino are themselves variable: the Spanish-derived counting system coexists with native Tagalog numerals. A reference that reads “100” and a hypothesis that reads isang daan are semantically identical. Under minimum edit distance, which is what WER is, that pair costs one substitution and one insertion, not three errors.

But every other class on the list shares a property that numerals lack: both forms are recoverable from the text alone. Kaniyang and kanyang are the same word spelled two ways, and a rule can canonicalise either direction without knowing anything about the audio. A digit is not like that. “20” before a noun is dalawampung; “alas 10” is alas diyes; “Setyembre 20” is a day-of-month reading; “F1” and ”20s” are not numerals at all. The reading was fixed by whoever spoke, and the digit does not record which one it was.

This makes expansion a guess in both directions. Expanding digits in the reference invents a reading the transcriber never committed to. Expanding them in the hypothesis discards a decision the recognizer actually made from the audio: the one piece of evidence that bears on the question.

So numerals are not a gap in the scorer. They are a defect in the reference, and they belong with the other things a reference fails to declare. The remedy is a reference that records the spoken form, not a rule that reconstructs it afterwards. In this project’s own normalization pipeline, numeral expansion is implemented and switched off by default for exactly this reason.

The cost is not small. Digit-bearing references make up close to a fifth of the FLEURS test set, and under normal scoring they carry a share of the measured error well out of proportion to their share of the utterances. A strong system’s error rate on them runs several times higher than on the rest. Current figures live on the leaderboard, not here, for the same reason the rest of this essay keeps them there.

The boundary is also blurrier than it looks. A count of digit-bearing references includes F1, M16, 1920s and 20s, which are digit-bearing but not numerals at all. The scoring mismatch does not care about that distinction. It only cares that the reference wrote a character the speaker did not say. That the same field mixes readable quantities, model designations and decade names is itself part of the problem.

The same defect reappears wherever a transcript is produced rather than scored, training labels included. There the fix already runs in daily practice, not just as a possibility. See the case study on silver-label generation at the end of this essay.

What the Numbers Look Like When You Change the Rules

The practical impact of normalization choices on Filipino ASR scores is not hypothetical. Using the evaluation infrastructure behind the FlipVox ASR Leaderboard, which stores raw model hypotheses separately from scored reports, it is possible to re-score the same model outputs under different normalization conditions without re-running inference.

When the same set of hypotheses is scored under a minimal normalization (lowercase and punctuation removal only) versus a normalization that handles the Filipino-specific equivalences described above, rankings between models shift. Systems that happen to match the reference transcriber’s orthographic preferences score better under minimal normalization, regardless of acoustic accuracy. Systems whose language model favors different but equally valid spellings are penalized.

The specific numbers belong on the leaderboard, not in this essay. The point is structural: the choice of normalization protocol is not a minor technical detail. It determines which model appears to be better, and that determination can reverse depending on rules that have nothing to do with whether the model understood what was said.

Parallel Efforts in Other Languages

Filipino is not the only language where scoring conventions are inadequate. Several recent efforts address analogous problems in other linguistic contexts, and each offers lessons.

The Open ASR Leaderboard (March 2026, 86 models, 12 datasets) standardizes text normalization across systems and open-sources all evaluation code, establishing the principle that normalization must be a declared, reproducible protocol rather than an undocumented preprocessing step. Its documented weakness is the asymmetry between English and non-English normalization, which is exactly the gap this essay describes for Filipino.

WERd (Ali et al., 2017) addresses dialectal Arabic, where the absence of standardized orthography means that multiple valid transcriptions exist for the same utterance. WERd defines equivalence classes for dialect-specific spelling variants, functionally equivalent to a GLM file for Arabic.

SN-WER (Script-Normalized WER, June 2026) tackles multi-script Indic ASR, where models trained on romanized text produce romanized hypotheses against native-script references. SN-WER normalizes through transliteration before scoring, separating script-mismatch errors from actual recognition errors. The underlying principle, that the scorer must distinguish orthographic convention differences from genuine mistakes, is identical to what Filipino scoring requires.

The Manohar and Pillai finding cited above is perhaps the most direct warning of all: when the field adopts a single normalization routine across languages, languages that routine was not designed for get evaluation results that do not reflect reality.

The Four Requirements

The gap is specific and fillable. Filipino ASR evaluation needs four things, and since this essay first published, two of them have moved.

First, a Filipino text normalization specification: a documented, versioned, publicly available set of rules for transforming both reference and hypothesis text into a canonical form before scoring. This specification must handle the orthographic variation classes described above, and it must be developed with input from Filipino linguists and transcription practitioners, not derived from English conventions. This one is still open.

Second, a Filipino GLM file (or its functional equivalent in whatever scoring tool the community adopts): a machine-readable encoding of the equivalence rules from the normalization specification, so that the scorer can treat legitimate variants as non-errors. This now exists. A Filipino GLM is in daily use behind the leaderboard: SCTK-format rules covering the classes described above, each grounded in Ortograpiyang Pambansa (KWF 2014) and checked against the KWF Diksiyonaryo. Every model score reported here is produced under it. It is a first draft by one practitioner, not a community standard, which is precisely why the first item still matters: a rule file without an agreed specification behind it is one person’s judgment, however carefully made.

Building it also surfaced a methodological point that generalizes past Filipino. Equivalence rules can be mined semi-automatically from the substitutions a scorer reports, but frequency and dictionary attestation alone are not enough to trust a proposal. Three classes had to be guarded explicitly, each after producing a real false positive: function words are never proposed, because folding a homophonous grammatical pair would erase genuine error signal rather than orthographic noise; proper nouns are routed to human review, because attestation inverts on names: a place name absent from the dictionary while a common word one letter away is present will “correct” the name into the wrong word; and doubled letters are never collapsed, because in Filipino a doubled vowel can mark aspect and a doubled consonant can come from prefix gemination, so the pair is morphology rather than a spelling variant. Anyone building an equivalence file for another low-resource language should expect the same three traps.

Third, a reference specification: the requirement this essay originally missed. A normalization rule can only canonicalise what the text already records. Where the reference has thrown information away, as it does with numerals, no amount of scoring machinery recovers it. Filipino needs transcription conventions that declare what the reference commits to: whether numerals record the spoken reading, whether disfluencies are transcribed, whether code-switched terms follow source or Filipino orthography. That is a decision made at annotation time, and it cannot be retrofitted onto a reference that already exists. It can be built into one that does not exist yet, which is what the case study below shows.

Fourth, a multi-condition reporting convention: the practice of reporting WER under at least two normalization conditions (a strict baseline and a linguistically informed normalization) so that readers can see how much of the reported error rate is attributable to orthographic convention rather than recognition failure. The FlipVox ASR Leaderboard adopts this practice.

None of these are research problems. They are engineering and community-coordination problems. The scoring infrastructure existed thirty years ago. The linguistic knowledge exists today. What is missing is the work of connecting them for Filipino.

Silver Labels: The Reference Specification You Can Write Today

The third requirement above is not only a wishlist item for some future reference. It already has a concrete answer for one specific case: transcripts produced by machine rather than by hand (pseudo-labels for training a model, rather than a reference for scoring one). There, the spoken form can be preserved, but only if it is requested at the point of transcription.

Every major cloud ASR API returns “display” text by default: punctuation, truecasing, and inverse text normalization, with numbers rendered as digits. That default is correct for scoring against a display-formatted reference. It is wrong for training data, because a label reading 1945 sits against audio that might have been spoken in Spanish series, in English, or in native Filipino counting, and the label has discarded precisely the distinction the acoustics carry. Scale makes that worse rather than better: more hours of an ambiguous mapping reinforce it.

The vendors differ more than their documentation suggests. Surveying six of them:

• One exposes spoken forms surgically: request entity metadata and each number carries the words it was said as, so ITN can be undone while punctuation and casing survive untouched. This is the only clean outcome.
• Two expose ITN as a parameter separable from punctuation, though one of them bundles casing in with it, so casing is lost as collateral.
• One returns no un-normalized text at all on its fast endpoint; the lexical form exists only on a different, slower API path.
• Two have no documented way to suppress ITN whatsoever.

Only the first genuinely satisfies the requirement. And the distinction matters for a second reason beyond the label itself: when several systems’ outputs are combined by voting, the votes are aligned on normalized text, and normalization strips case and punctuation but leaves digits alone. A system emitting 1945 and one emitting the spoken words therefore do not reinforce each other at that position. They enter as unrelated tokens and split the vote, at exactly the positions already measured as the hardest.

A single clip from a Tagalog broadcast corpus makes the stakes concrete. In a passage about a number of teachers, the speaker appears to self-correct mid-number: apatnapu’t libo… apatnapu’t limang libo (forty thousand… forty-five thousand). Of thirteen systems evaluated against that corpus, spanning transducer, cloud and encoder-decoder architectures from different vendors, nine independently produced the same repetition: zipformer, Azure, Vosk, Deepgram, Speechmatics, Qwen, OWSM, SeamlessM4T and Chirp 3. I have not listened to the clip myself, so this is not a verified transcript of what was said. It is nine independent systems agreeing on the same repetition, and agreement at that scale is evidence of speech, not decoding error. The verbatim composite kept it: isang daan at apatnapu’t na apatnapu’t limang libong. The display composite, which applies ITN, produced isandaan at limang libong instead: a fluent, well-formed 105,000 that no system transcribed and nobody said, with the disfluency gone. A model trained on the display label would learn a mapping from self-repair audio to text with no repair in it, at exactly the frames hardest to align.

The practical rule that follows is narrower than “turn off formatting.” Turn off inverse text normalization only. Punctuation and casing are removable later without loss; a spoken form, once replaced by a digit, is not recoverable at all.