How Filipino ASR Should Be Scored

The Problem in One Sentence

When you evaluate a Filipino ASR system using standard tools and default settings, a significant portion of what the scorer reports as errors are not recognition failures. They are disagreements about how to spell the same word.

This essay explains how that happens, why the field’s standard evaluation tools lack the machinery to prevent it, and what a proper Filipino scoring protocol would look like.

What WER Assumes

Word Error Rate, the dominant metric in speech recognition evaluation since the 1990s, is computed by aligning a reference transcript against a hypothesis (the system’s output) and counting substitutions, deletions, and insertions. The formula is simple: the sum of those three error types, divided by the number of words in the reference.

The simplicity is deceptive. WER assumes that the reference and the hypothesis share the same orthographic conventions. If the reference spells a word one way and the system produces a legitimate alternative spelling, WER counts it as a substitution. If the reference includes a hesitation marker and the system omits it, WER counts a deletion. If the reference writes a number as digits and the system spells it out, every word in the spelled-out form becomes an insertion or substitution. None of these are recognition failures. All of them inflate the error rate.

For English, this problem is manageable because English orthography is relatively standardized. For Filipino, it is not.

What the Field Built and Then Forgot

The mismatch between what a model produces and how a reference represents the same speech is not a new problem. It was identified and solved, procedurally, in the 1990s.

The NIST Speech Technology Evaluation program, which ran the Hub-4 broadcast news and Switchboard evaluations that shaped a generation of ASR research, developed a scoring toolkit called SCTK (the NIST Scoring Toolkit). Its core program, sclite, implemented Levenshtein alignment, the same dynamic programming algorithm behind WER. But sclite did not stop at alignment. It supported a rich set of mechanisms for handling legitimate variation between reference and hypothesis.

GLM files (Global Language Mapping) allowed evaluators to define equivalence classes before scoring. Contractions could be mapped to their expanded forms, spelling variants could be declared equivalent, and the resulting alternations were embedded directly into the reference text. When sclite encountered a word that had been mapped to multiple acceptable forms, it chose the alignment that minimized the error count. The scorer knew, because a human had told it, that “gonna” and “going to” were the same utterance.

STM format (Segment Time Mark) supported inline alternations using a {text / alt} syntax, a null-word symbol whose insertion was not counted as an error, and words marked optional with parentheses. Optional-word scoring treated marked reference or hypothesis words as optionally deletable, so that a hesitation filler in the reference would not penalize a system that (correctly) omitted it.

CTM format (Conversation Time Mark) carried ALT blocks representing alternative decodings, typically created by GLM filtering.

Disfluency protocols were explicit. In the Hub-4 evaluations, pause fillers were removed from references via a filter script before scoring. Overlapping-speech regions were tagged and excluded from scoring entirely. These were not informal conventions. They were published evaluation plans that participating labs agreed to follow.

The entire architecture reflected a basic insight: the question “did the system get it right?” cannot be answered by string matching alone. It requires a definition of what counts as equivalent, and that definition must be encoded somewhere the scorer can reach it.

This infrastructure was not perfect. sclite’s C source has known bugs, including one where doubly-nested alternatives after GLM filtering fail to expand correctly. Researchers as recently as 2022 were patching the original C code to work around these issues. No faithful Python reimplementation of sclite exists.

But the conceptual framework was sound, and for twenty years it was standard practice.

What Replaced It

The end-to-end revolution in ASR, beginning roughly with Deep Speech in 2014 and accelerating through wav2vec 2.0, HuBERT, and Whisper, changed not just how models were built but how they were evaluated. The new models output UTF-8 text directly. The old Viterbi-decoded lattices and word graphs are gone. And with them, the ecosystem of scoring tools that operated on those structured representations largely disappeared from common practice.

The replacement, in most published work and open-source pipelines, is a single function call. In Python, the dominant package is jiwer, which computes WER in one line:

wer = jiwer.wer(reference, hypothesis)

jiwer performs basic text normalization: lowercasing, punctuation removal, whitespace collapsing. It does not support alternations. It does not support optional words. It does not support GLM-style equivalence mappings. It does not have a concept of language-specific normalization rules. It computes Levenshtein distance on two strings and returns a number.

This is not a criticism of jiwer as software. It does what it was designed to do. The problem is that it is now the standard evaluation tool for languages whose orthography demands exactly the scoring flexibility that jiwer does not provide.

Whisper made this worse by establishing a normalization convention that the rest of the field adopted. Whisper ships with two text normalizers: an English normalizer that performs rich, linguistically informed transformations (expanding contractions, normalizing numbers, handling currencies and abbreviations), and a “basic” normalizer for all other languages that strips punctuation, lowercases, and removes Unicode characters in the mark class.

That last operation is catastrophic for languages written in Indic scripts. Manohar and Pillai (EMNLP 2024, “What is lost in Normalization?”) demonstrated that Whisper’s normalization removes vowel signs (matras) from Indic scripts, producing artificial WER reductions between 10 and 34 percentage points. The same normalization routine has been adopted by MMS, SeamlessM4T, and AssemblyAI. A model that destroys vowel signs and then reports low WER under a normalizer that also destroys vowel signs has not demonstrated accurate transcription. It has demonstrated consistent destruction.

Filipino, written in Latin script, does not suffer from the vowel-sign problem. But it suffers from the same underlying asymmetry: the English normalizer was built by people who understand English; the multilingual normalizer was built by people who needed something that would not crash on non-English text. Nobody wrote the normalizer that knows Filipino.

What Filipino Needs

Filipino’s orthographic variation falls into several classes, each of which would be a rule or a set of rules in a GLM file. The examples below are drawn from a systematic audit of the FLEURS fil_ph evaluation set (documented separately in Problems with FLEURS as a Filipino Benchmark).

Clitic and affix boundary variation. Filipino has enclitic particles and ligatures that can be written as part of the preceding word or as separate tokens. The most common case is the linker na/ng, which appears as -ng suffixed to vowel-final words (e.g., malaking from malaki + na) or as a separate word na after consonant-final words. But the boundary is not always observed consistently, and compounds involving din/rin, daw/raw, and lang/lamang can be written as one word or two. A GLM rule would declare these forms equivalent.

Specific high-frequency examples from the FLEURS audit:

The ng/nang pair deserves particular attention. These two words are homophonous in spoken Filipino: both are pronounced /naŋ/. The distinction is purely orthographic (and, for careful writers, grammatical: ng marks a nominal complement, nang marks an adverbial or temporal modifier). A model that outputs ng where the reference has nang, or vice versa, has not made an acoustic error. It has made an orthographic choice that even native speakers frequently disagree on. Any scoring treatment of this pair is a policy decision, not a ground truth.

Code-switched loanword spelling. Filipino speech routinely incorporates English words, Spanish-derived vocabulary, and terms from other Philippine languages. The orthographic treatment of these words is unstandardized. “Computer” might appear as kompyuter, computer, or kumpyuter depending on the transcriber’s preference and the register of the text. Spanish-derived words like presidente coexist with the Filipinized pangulo. A GLM rule cannot enumerate every possible loanword variant, but it can handle the high-frequency cases that dominate error counts.

Numeral rendering. Numbers can be written as digits or spelled out, and the spelled-out forms in Filipino are themselves variable (the Spanish-derived counting system coexists with native Tagalog numerals for numbers below twenty). A reference that reads “100” and a hypothesis that reads “isang daan” are semantically identical. Under default WER scoring, this single number generates one deletion (the digit) and two insertions (the two words), inflating the error count by three.

Disfluency handling. Conversational Filipino, like any natural speech, includes hesitation fillers (ah, eh, ano, kasi), false starts, and repairs. Some of these (particularly ano and kasi) are also content words in other contexts, making automatic removal risky. A scoring protocol needs explicit rules for which fillers to exclude and under what conditions, not a blanket strip-all-hesitations approach.

What the Numbers Look Like When You Change the Rules

The practical impact of normalization choices on Filipino ASR scores is not hypothetical. Using the evaluation infrastructure behind the FlipVox ASR Leaderboard, which stores raw model hypotheses separately from scored reports, it is possible to re-score the same model outputs under different normalization conditions without re-running inference.

When the same set of hypotheses is scored under a minimal normalization (lowercase and punctuation removal only) versus a normalization that handles the Filipino-specific equivalences described above, rankings between models shift. Systems that happen to match the reference transcriber’s orthographic preferences score better under minimal normalization, regardless of acoustic accuracy. Systems whose language model favors different but equally valid spellings are penalized.

The specific numbers belong on the leaderboard, not in this essay. The point is structural: the choice of normalization protocol is not a minor technical detail. It determines which model appears to be better, and that determination can reverse depending on rules that have nothing to do with whether the model understood what was said.

Parallel Efforts in Other Languages

Filipino is not the only language where scoring conventions are inadequate. Several recent efforts address analogous problems in other linguistic contexts, and each offers lessons.

The Open ASR Leaderboard (March 2026, 86 models, 12 datasets) standardizes text normalization across systems and open-sources all evaluation code, establishing the principle that normalization must be a declared, reproducible protocol rather than an undocumented preprocessing step. Its documented weakness is the asymmetry between English and non-English normalization, which is exactly the gap this essay describes for Filipino.

WERd (Ali et al., 2017) addresses dialectal Arabic, where the absence of standardized orthography means that multiple valid transcriptions exist for the same utterance. WERd defines equivalence classes for dialect-specific spelling variants, functionally equivalent to a GLM file for Arabic.

SN-WER (Script-Normalized WER, June 2026) tackles multi-script Indic ASR, where models trained on romanized text produce romanized hypotheses against native-script references. SN-WER normalizes through transliteration before scoring, separating script-mismatch errors from actual recognition errors. The underlying principle, that the scorer must distinguish orthographic convention differences from genuine mistakes, is identical to what Filipino scoring requires.

The Manohar and Pillai finding (EMNLP 2024) that Whisper’s normalizer inflates Indic language scores by 10 to 34 percentage points is perhaps the most direct warning: when the field adopts a single normalization routine across languages, languages that the routine was not designed for get evaluation results that do not reflect reality.

What Needs to Happen

The gap is specific and fillable. Filipino ASR evaluation needs three things that do not currently exist.

First, a Filipino text normalization specification: a documented, versioned, publicly available set of rules for transforming both reference and hypothesis text into a canonical form before scoring. This specification must handle the orthographic variation classes described above, and it must be developed with input from Filipino linguists and transcription practitioners, not derived from English conventions.

Second, a Filipino GLM file (or its functional equivalent in whatever scoring tool the community adopts): a machine-readable encoding of the equivalence rules from the normalization specification, so that the scorer can treat legitimate variants as non-errors.

Third, a multi-condition reporting convention: the practice of reporting WER under at least two normalization conditions (a strict baseline and a linguistically informed normalization) so that readers can see how much of the reported error rate is attributable to orthographic convention rather than recognition failure. The FlipVox ASR Leaderboard adopts this practice.

None of these are research problems. They are engineering and community-coordination problems. The scoring infrastructure existed thirty years ago. The linguistic knowledge exists today. What is missing is the work of connecting them for Filipino.

This essay is part of a series on Filipino ASR evaluation. See also: The Landscape of Filipino Speech Datasets and Problems with FLEURS as a Filipino Benchmark. The methodology described here informs the FlipVox ASR Leaderboard and a forthcoming arXiv paper on Filipino ASR evaluation.