UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding

The critical methodological choice here is sourcing questions from authentic Pakistani and Indian educational materials rather than translating existing English benchmarks, which eliminates the translation artifacts that typically inflate scores and obscure genuine comprehension failures. That design decision is what makes the performance gaps UrduMMLU surfaces credible rather than noise.

This fits directly into a pattern Modelwire has been tracking across multiple recent papers. K-BrowseComp (June 1) showed frontier models dropping 30-45 percentage points on Korean web tasks compared to English equivalents, and UrduMMLU is now producing analogous disparity findings for Urdu at a much larger subject-coverage scale. The FRANZ audit framework covered here on June 1 adds another layer: even where models score adequately on factual benchmarks, communicative framing in non-English contexts remains unaudited. Together these papers suggest the evaluation gap for non-English languages is structural, not incidental, and that no single benchmark type captures it fully.

Watch whether the teams behind leading multilingual models (Gemini-3 is already tested here) publish targeted fine-tuning or retrieval-augmented results on UrduMMLU within the next six months. Sustained low scores with no response from major labs would confirm that Urdu remains a genuine capability blind spot rather than a fixable data gap.