Multilingual self-play reveals language-dependent skill gaps in LLMs

The paper isolates language as a confound variable in capability measurement. Prior work has shown models fail to generalize across domains (fact-checking systems, cough detection), but this work asks whether performance gaps within a single task stem from linguistic encoding or actual skill deficits. That distinction changes how you interpret multilingual benchmark results.

This connects directly to the cross-dataset evaluation pattern we've covered repeatedly. The fact-checking systems paper from August showed fine-tuned models fail to transfer across domains; the TB cough work revealed models cluster by device rather than disease signal; the KPA benchmark piece exposed how flawed evaluation masks real performance ceilings. This language-invariance work applies the same diagnostic lens to a new axis: it's asking whether multilingual LLM benchmarks are measuring skill or just measuring which languages received adequate training data. The omitted variable here is linguistic representation quality, not domain shift or data artifacts, but the methodological problem is identical.

If researchers run the same competitive self-play framework on low-resource languages (Swahili, Tagalog, Icelandic) and find performance gaps persist even when controlling for training data volume, that confirms language encoding is the bottleneck. If gaps disappear once you account for pretraining corpus size per language, the story becomes about data coverage, not a fundamental skill invariance problem.