I was designing a correction pass for proper nouns in transcription labels when one prompt stopped me. The plan was sensible enough: mine the capitalised spans, group the variants that share context, resolve each group against a gazetteer of known Philippine names. Then I looked at what the grouping actually collected, and the plan stopped being the interesting part.
The corpus is a set of Filipino read speech. One prompt in it, roughly “Doon ko nakita si Dawn Zulueta noong piyesta. Maganda sa Hollywood showbiz.”, is read by 18 speakers. Everyone is looking at the same text. FlipVox ASR gave “Don” for the first name all 18 times. For the surname it gave six different spellings: Zuleta nine times, Solueta three, Zulweta twice, and Sulweta, Silueta and Zalweta once each. None of them is Zulueta. A majority vote over those outputs would pick Zuleta, with confidence, and be wrong.
The same system writes “Dawn Zulueta” correctly elsewhere in the warehouse. I found four rows of entertainment-news speech where it does, and those rows are full of showbiz talk: “Richard Gomez”, “love team”. So the model can produce the right string. What it did not do here was get there from the sound.
What the grouping showed me
The gazetteer fixes this. Group the six variants by their shared neighbours, look up the cluster, find Zulueta. That works. But it works because I bolted a step onto the outside that the model does not have inside. I keep coming back to that.
Here is what I think happened, and I want to be clear that it is my reading of the outputs and not something I know about how Zipformer works. In the news rows, the words around the name told the model who was being discussed, and the name followed. In the read prompt, the context is a festival and Hollywood and a vague showbiz sentence. That is not enough, so the model fell back on the sound. And the sound is different for each of the 18 people.
The question I can’t put down
The 18 speakers are not doing anything wrong. They are reading a prompt, and each of them says it the way they say it. One rounds the ue, one glides it into a w, one swallows a syllable. Some of that is region and some is how a particular mouth works. Some of it is probably a person reading a name quickly off a page. I can’t tell which from the transcripts, and I’d want to listen before saying more.
What I keep asking is this. When I read “Dawn Zulueta” I am not decoding the sound. I know who she is, I know the sentence is about running into a famous actress, and my brain meets the speaker halfway. The pronunciation can be off and I still land on the right name, because the name is what I expected. An ASR model is asked to do the same thing, and I don’t think it gets a separate place to do it.
My own framing, for what it is worth: the older pipelines had a pronunciation lexicon sitting between the acoustic model and the words. A word could carry several accepted pronunciations, and that was one place where “people say this differently” got written down. The end-to-end systems I evaluate go from audio straight to spelling. I don’t know what, if anything, plays that lexicon’s role inside them, and I haven’t verified how every moving part is built, so I’m not going to claim it has no such layer. I’m only saying that the outputs above look like a system that is missing one.
There are really two jobs in there, and they get run together. One is what was said: a mispronounced, clipped, accented, tired, congested version of a name. The other is what was meant: Dawn Zulueta, the actress. The second needs world knowledge and some sense of the speaker’s intent. A model asked to do both in one pass will do the second well when the context hands it the answer, and fall back on the first when it doesn’t. The Zulueta rows look like that second case to me.
Why this matters for labels
It also changes how I think about the correction pass. If grouping by context and resolving against a gazetteer works, I am doing the meant-step by hand, after the fact, on text that came out of the said-step. That is fine for a silver label, as long as I’m honest that it is a repair and not a measurement of what the speaker produced. A label that says “Zulueta” for Sulweta is the right spelling and not a record of the audio. Those two things are different, and a corpus can be good at one and silent on the other.
It also means the aggregate can hide this. Eighteen rows is nothing in a corpus this size, and a WER figure would not move. I only saw it because one prompt repeated across speakers, which is the one situation where you get to hold the text fixed and watch the audio vary. That is a rare setup. Most corpora never give you the control.
Where I am
I don’t have an answer. I don’t know how a model should account for mispronunciation, disfluency, or the physical facts of a particular voice without either ignoring the audio or inventing words nobody said. Lean too hard on context and it rewrites speakers into what it expected. Lean too hard on the sound and you get Sulweta.
So the question stays open, and I’d rather leave it that way than pretend the gazetteer closes it: what, inside an ASR system, is supposed to know that eighteen different sounds were all the same name?