Iāve been building Filipino silver labels the slow way: run a number of ASR systems over a corpus, combine their outputs with NIST SCTKās ROVER, and take the result as the label. Today I was reading through the combined output line by line and found the exact failure Iād been half-expecting.
One reference sentence includes the word impluwensiyang, āimpluwensiyaā (influence) with the -ng linker before the following adjective. Several systems disagreed on how to spell it. One dropped the w partway through. Another kept the w but dropped an i further along. A third matched the reference exactly. The two near-miss spellings, each wrong in its own small way, outvoted the one correct spelling between them. What came out of the combination was neither the referenceās spelling nor a deliberate choice of any kind, just whichever near-miss happened to have more company.
The same word, combined from a different set of systems, came out right, because enough of them happened to agree on the exact spelling. So on this one word I have both outcomes sitting side by side: fragmentation that broke the result, and fragmentation that didnāt, from the same reference word.
I found a cleaner case a few sentences later, on the word for diameter. Four plausible spellings showed up across the raw outputs, differing from each other by one or two characters and from the referenceās own spelling by about the same. The combination process split them into their own separate camps instead of recognizing them as competing guesses at the same thing, and what came out the other end didnāt match how the reference had spelled it either.
Worth being honest that this doesnāt always break. A few sentences on, the word for sanctuary gets rendered several different ways by a handful of systems, but the rest land on the exact reference spelling, so the combined result comes out correct anyway. Fragmentation is a threat in proportion to how close the split is, not a guarantee of failure every time it shows up. I donāt yet have a good sense of how often this happens across everything Iāve combined so far, only that it isnāt rare.
How the combination actually works
Worth being precise about what makes this fragmentation possible, without the particulars of my own setup. Every systemās output gets normalized the same way before anything else happens: lowercased, punctuation handled consistently. Then itās a frequency vote: a word-level alignment folds every systemās output into a shared skeleton, and whichever word appears most often at a given position wins that slot.
So the alignment that decides the outcome is a word-level dynamic program, and that program treats impluensyang and impluwensiyang as two completely unrelated tokens, the same as if one of them had been an entirely different word. Substitution cost is flat: 1 if the words donāt match exactly, 0 if they do. One character apart costs the same as sharing nothing.
The wrong shape first
My first instinct was to cluster the variants before combining: bucket the near-miss spellings into one canonical form as a preprocessing pass, then hand the cleaned-up outputs to the combination step. I sat with that for about an hour before deciding itās the wrong shape. Itās a patch on the outside of the alignment, a separate stage that has to guess at clusters independently of the thing that actually decides correspondences. If the clustering gets a word wrong, or misses a variant, thereās no way for the alignment itself to notice or correct for it. It also doesnāt touch the deeper issue, which is that the DPās notion of ādifferent wordā is binary when it doesnāt need to be.
The idea
What I actually want is a nested alignment: the outer DP still aligns word sequences the way it does now, but the substitution cost between two words isnāt 0-or-1 anymore. Itās the character-level edit distance between them, normalized by the length of the longer word. Take that cluster from today: impluensyang to impluwensyang is one insertion, cost 1/13 ā 0.08; impluwensyang to impluwensiyang is one insertion, cost 1/14 ā 0.07; impluensyang to the correct impluwensiyang is two insertions, cost 2/14 ā 0.14. All three sit close together and close to zero. Diametro to diyametro is one insertion, cost 1/9 ā 0.11. Compare any of those to an unrelated word and the cost sits near the maximum. Near-variants stop looking like unrelated tokens to the alignment and start looking like what they are: cheap corrections of each other.
This doesnāt just change who wins inside a slot. It changes where the correspondences fall in the first place, which is the part I actually care about. It should also absorb split/merge cases for free, things like nagsasaad against nag sasaad, where a binary-cost word DP has no cheap move and has to eat a full insertion penalty for whatās really a spacing difference.
Whatās already out there
I did a few searches today, not a literature review, so take the negative result as weak evidence and nothing stronger.
The character-aware substitution cost itself is not new. texterrors, a scoring tool by Rudolf Braun, does exactly this: substitution cost is the character edit distance between two words divided by the length of the longer one, which is the same [0,1] normalization I landed on independently by working through todayās examples. It ships a -no-chardiff flag to fall back to plain Kaldi-style alignment, which tells you character-aware is the deliberate default, not an afterthought. Braun presents it as his own fix in a blog post and doesnāt cite prior work for the technique specifically: https://ruabraun.github.io/jekyll/update/2020/11/27/On-word-error-rates.html. Thereās also recent work on word-level error analysis at Interspeech 2025 worth reading alongside it: https://www.isca-archive.org/interspeech_2025/huang25j_interspeech.html
What I couldnāt find is the same idea inside system combination. ROVER and confusion-network combination still build the word transition network with plain Levenshtein, first hypothesis as the skeleton, each subsequent one aligned in on top (see NISTās own documentation: https://github.com/usnistgov/SCTK/blob/master/doc/rover/rover.htm). The published variants I found all target something else. Multistage ROVER is about efficiency, not alignment quality (https://www.researchgate.net/publication/221264976_An_efficient_multistage_ROVER_for_automatic_speech_recognition). Quality-estimation ROVER reorders which hypothesis goes in first (https://arxiv.org/pdf/1706.07238). LV-ROVER adds a lexicon-verification stage on top of the vote (https://arxiv.org/pdf/1707.07432). MOVER extends the approach to meetings (https://arxiv.org/html/2508.05055). None of them touch the substitution cost inside the DP itself. Again: three searches, not exhaustive. Iād rather be wrong about this being unclaimed than claim it and get corrected later.
If the gap really is there, I think itās because the two use cases have historically had different tolerances. In scoring, a character-aware cost changes how an error gets reported after the fact. In combination, it changes which words end up in the same slot in the transition network, which means it changes what the combined result actually says, as directly as the impluensyang case shows. Thatās a bigger blast radius, and probably why itās had less appetite behind it.
A prediction I can test
ROVERās multi-way alignment is greedy: it grows a combined transcript from one hypothesis and folds each subsequent one in, so the result can depend on the order systems are combined in. If a character-aware cost is doing its job, near-variants should align to each other regardless of that order, which means the result should get less sensitive to it as the cost gets more graded. Thatās checkable against combinations Iāve already built, not something Iād need new data for.
Alignment Is Not Equivalence
Sitting with the Filipino morphology pairs longer, I realized Iād been asking them to answer a question they donāt actually bear on. There are two separate questions buried in this idea, and Iād been treating them as one.
The first is alignment: do two words occupy the same stretch of audio, so they belong in the same slot of the network as competing guesses about what was said there? The second is equivalence: assuming they do, are they actually the same word, so their votes should be merged into one candidate? A graded substitution cost only answers the first question. It has nothing to say about the second.
Once I separated them, the morphology pairs stopped looking like a threat to the idea and turned into exactly the reason the distinction matters. Ng and nang are near-homophones. If one system says one and another says the other, they are almost certainly two guesses about the same stretch of audio, and getting them to align, landing in the same slot instead of drifting apart and dragging their neighbors into false substitutions around them, is the correct outcome, not a risk. The danger was never in aligning them. Itās in what happens after: whether the vote then treats them as interchangeable and folds them into one candidate. That has to stay no. Same shape for umabot against umaabot, for galaw against paggalawin: close enough to belong in the same slot, wrong to become the same word once theyāre there.
So the fix isnāt a smarter distance function that knows to leave those pairs alone. Itās not asking one number to carry two answers. Alignment is a distance question. Equivalence is a linguistic one, and it belongs downstream of the alignment, not baked into it.
Thereās a better answer to the alignment question sitting right there, and it doesnāt need spelling at all. If the word timings coming out of each system were real instead of fabricated, time overlap would settle whether two words are competing for the same slot far more directly than how similar their spelling is: two words spoken over the same half-second are competing hypotheses regardless of how many letters they share. Thatās what time-mediated alignment exists for in the standard tooling, and itās the more honest signal. It isnāt a perfect one, though. Timings are themselves model estimates, and systems disagree with each other at word boundaries and around insertions and deletions, so time overlap is better evidence, not ground truth. And it still only answers alignment, not equivalence: whether two time-overlapping words should merge is exactly the same downstream linguistic call either way.
Today I donāt have real per-word timings to feed into anything, so what I have in place of them is a placeholder, and time-mediated alignment isnāt actually usable yet. Character distance is the proxy available while thatās true. But I should be honest about what itās a proxy for: not a judgment on whether two words are the same, just a rough stand-in for whether they were probably said at the same moment. Once real timings exist, distance should shrink to a tiebreak for wherever timing is missing or unreliable, not stay the primary signal.