Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number Game

The key methodological move here is obfuscation as a control condition: by replacing standard identifiers with arbitrary symbols, the researchers remove the possibility that a model is retrieving proof patterns from training data rather than constructing them. The performance drop is the signal, not the absolute score.

This fits into a cluster of papers arriving simultaneously that all point at the same underlying problem from different angles. The ARC-AGI-3 analysis covered here ('Even the latest AI models make three systematic reasoning errors') found repeatable failure modes that persist regardless of scale, and the Obfuscated Natural Number Game result is structurally similar: both isolate conditions where surface familiarity is removed and model performance collapses. The procedural execution study from arXiv cs.CL on the same date adds a third data point, showing accuracy falling from 61% to 20% as task length grows. None of these papers are coordinated, but together they form a consistent picture: benchmark scores in formal reasoning are partly a measure of training distribution overlap, not generalizable inference capacity.

Watch whether any of the major theorem-proving labs (Lean community contributors, DeepMind's AlphaProof team) attempt to replicate the obfuscation methodology on their own eval suites within the next two quarters. Adoption of obfuscation as a standard control would signal the field accepts the contamination critique; silence would suggest the result is being treated as an outlier.