Compression-based evaluation framework targets data contamination in LLM benchmarks
Researchers propose a novel evaluation framework that sidesteps data contamination by measuring how well language models compress newly published text. Rather than relying on static benchmarks that models may have encountered during pretraining, Uncheatable Eval dynamically sources fresh material across 14 text categories and uses compression rate as a proxy for predictive capability. The approach addresses a critical blind spot in base model assessment, where instruction-following limitations already complicate traditional task-based testing. Testing 80 models reveals how compression performance scales with context length, offering practitioners a contamination-resistant signal for comparing model quality as datasets grow and overlap becomes harder to track.
Modelwire context
ExplainerThe key insight is that compression rate works as a contamination-resistant proxy for model capability precisely because it sidesteps the benchmark-memorization problem. But the paper doesn't claim compression is a replacement for task-based evals; it's a supplementary signal that works because it measures something fundamentally different: how well a model predicts unseen token sequences rather than how well it solves curated problems.
This connects directly to the evaluation robustness concerns raised in recent coverage. The FakeContextBench paper from late September showed that standard pretraining leaves models vulnerable to misleading context, and the MWE-ECL work revealed that context-window scaling alone won't solve reasoning failures where models have strong priors. Uncheatable Eval addresses a related but distinct problem: it assumes the benchmark itself is compromised, not just the model's reasoning within it. Where those papers focused on what models do with information they receive, this one focuses on whether we can even trust that we're measuring genuine capability rather than memorization.
If the 80-model compression rankings diverge significantly from standard leaderboard orderings (MMLU, GSM8K) for models trained after mid-2025, that confirms compression captures something real about capability that benchmarks miss. Conversely, if compression rankings converge with existing benchmarks, the method becomes a consistency check rather than a new signal.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsUncheatable Eval
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.