XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics

The core problem XQ-MEval targets is subtle but consequential: existing translation metrics may score the same quality of error differently depending on which language pair is being evaluated, meaning a metric that looks reliable on English-German may quietly underperform on lower-resource pairs without anyone noticing.

This lands squarely in a cluster of benchmark-reliability concerns that dominated coverage on April 16. The 'Context Over Content' paper on LLM judges found that automated evaluators respond to contextual framing rather than actual output quality, and the 'Diagnosing LLM Judge Reliability' piece showed that aggregate consistency scores can mask per-instance logical failures in roughly one-third to two-thirds of documents. XQ-MEval raises an analogous concern one layer down: if the metrics used to train and select translation systems are themselves biased by language pair, then every downstream evaluation built on those metrics inherits that distortion. The 'Fabricator or dynamic translator' paper on MT hallucinations is also adjacent, since detecting spurious output depends on having reliable quality signals in the first place.

Watch whether MQM-based metrics show measurable score variance across the nine language pairs when applied to XQ-MEval's injected-error set. If the variance is large on low-resource pairs but small on high-resource ones, that confirms the bias is systematic rather than noise.