Automatic metrics miss quality gaps in romanized Indic LLM outputs

The study isolates a failure mode that automatic metrics themselves cannot see: they produce confident scores that diverge from human judgment, making the problem invisible to practitioners who rely on leaderboards. This is distinct from showing that metrics are imperfect; it shows they actively mask real performance differences in specific linguistic contexts.

This finding extends a pattern established across recent coverage. Like the Khmer benchmark work from earlier this week, it exposes how evaluation infrastructure built for high-resource languages fails silently on underrepresented variants. But where Khmer PFGen-Bench proposed a reference-based workaround, this study reveals the deeper problem: even when you run the same automatic metrics on the same outputs, they produce systematically misleading signals. This connects directly to the broader audit work on scoring instability (the preregistered judge study and evaluation contracts papers from last week), which showed that benchmark verdicts are fragile artifacts of design choices rather than robust capability signals. The difference here is that the fragility isn't accidental; it's baked into metrics optimized for English-centric training data.

If Claude, GPT-5.5, Gemini, or Qwen release updated romanized Indic benchmarks in the next two quarters that weight human judgment equally with automatic metrics, that signals the finding has moved from research to practice. Conversely, if these models continue shipping romanized Indic capabilities without updating their evaluation pipelines, it confirms that leaderboard pressure outweighs the evidence.