Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India

The district-level performance gap is the buried finding here: it suggests that existing Indic ASR models don't just struggle with language diversity but with intra-language regional variation, meaning a model trained on Mumbai Hindi may fail systematically on Bhojpuri-inflected Hindi from eastern UP. That granularity is absent from most multilingual speech benchmarks.

The benchmark design philosophy here mirrors what researchers did with MADE (covered April 16), which also prioritized real-world messiness over clean held-out splits, specifically to surface failure modes that curated datasets hide. Both papers are pushing back against the same tendency in ML evaluation: optimizing for leaderboard scores on scripted, controlled data while real deployment conditions look nothing like that. The Voice of India work is largely disconnected from the TTS and generative speech coverage in our archive, including the Gemini 3.1 Flash TTS release from April 15. That work is about synthesis expressiveness; this is about recognition robustness, and the two communities rarely share evaluation infrastructure.

Watch whether Whisper, MMS, or any of the major multilingual ASR systems publish results against this benchmark within six months. If none do, it signals the benchmark hasn't achieved the adoption needed to pressure vendors into closing the regional performance gaps it documents.