Evaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric Reliability

The more pointed finding here is not that Fongbe scores are low, but that BLEU and chrF++ scores actively mislead practitioners by failing to correlate with native-speaker judgment, meaning teams could ship a broken Fongbe translation system while their dashboards show acceptable numbers.

This connects directly to two threads in recent coverage. The ASR corpus work on Fongbe and Hausa (published the same day) addresses the upstream data scarcity problem, but this paper shows that even when you get models to produce output, you may lack the tools to know whether that output is any good. Separately, BabelJudge, covered a day later, documents the same class of failure from a different angle: LLM-as-a-judge systems collapse in non-English contexts, which compounds the metric unreliability problem this paper identifies. Together, the three papers sketch a compounding gap: low-resource languages lack training data, lack reliable automatic metrics, and lack judges capable of evaluating them.

Watch whether any of the four benchmarked models releases updated multilingual evaluation cards that include Fongbe specifically. If none do within six months, that is a reasonable signal that low-resource African language coverage remains a compliance checkbox rather than a development priority.