ASR family bias undermines text-to-speech evaluation metrics

Illustration accompanying: Best-of-$N$ TTS Evaluation is Confounded by ASR Family Alignment

Researchers uncovered a critical flaw in how text-to-speech systems are evaluated using best-of-N inference with ASR verifiers. The problem: ranking of candidate utterances flips dramatically depending on which ASR model judges them, even when those models share nearly identical internal representations. This reveals that evaluation metrics are coupled to ASR family lineage rather than genuine quality differences, potentially invalidating published TTS benchmarks. The team proposes cross-family ensemble ranking to mitigate the confound, signaling that current leaderboards may systematically favor verifier-evaluator pairs from the same vendor ecosystem.

Modelwire context

Skeptical read

The paper doesn't just identify that ASR choice affects TTS rankings. It shows the rankings flip entirely depending on ASR family, meaning published leaderboards may be measuring ASR-TTS compatibility rather than actual synthesis quality. The confound isn't noise; it's systematic bias baked into how the field validates progress.

This connects directly to the evaluation methodology problem surfaced in the LLM-as-judge personality recognition paper from the same day. Both papers expose how the choice of judge (ASR model here, LLM framework there) doesn't just introduce variance; it fundamentally alters which systems rank highest. The personality work showed that theory-dependent evaluation fragments results across competing schemas. Here, vendor-dependent evaluation fragments TTS rankings across ASR lineages. Both papers argue the field has mistaken evaluator choice for ground truth.

If F5-TTS and Whisper-based rankings remain stable when re-evaluated using the proposed cross-family ensemble on the same LibriSpeech-PC test set within six months, the mitigation works. If rankings still flip significantly even under ensemble voting, the confound runs deeper than ASR family alignment and the paper's solution is incomplete.

Coverage we drew on

Large-Language-Models-as-a-Judge in Theory-Agnostic Adaptive Metric-Alignment for Prototypical Networks in Personality Recognition · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsF5-TTS · Whisper · wav2vec 2.0 · HuBERT · LibriSpeech-PC

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Best-of-$N$ TTS Evaluation is Confounded by ASR Family Alignment”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.