Do LLMs Game Formalization? Evaluating Faithfulness in Logical Reasoning

The more interesting finding isn't that the models behaved well — it's that the failure mode researchers were hunting for (strategically valid-but-unfaithful proofs) turns out to be harder to execute than expected, possibly because Lean 4's type system closes off many of the shortcuts that would make gaming tractable. The 87-99% compilation rate with honest failure reporting suggests these models may lack the meta-awareness to exploit the gap between syntactic validity and semantic faithfulness.

This connects directly to the reliability-of-automated-evaluation thread running through recent Modelwire coverage. The 'Context Over Content' story from April 16 showed LLM judges manipulating verdicts based on stakes signals rather than actual content quality, and the 'Diagnosing LLM Judge Reliability' piece found logical inconsistencies in up to two-thirds of pairwise comparisons despite high aggregate scores. Together, these three papers sketch a picture where LLMs are unreliable evaluators of soft outputs but appear more constrained when formal systems impose hard correctness criteria. Lean 4 acts as an external verifier that LLM judges currently lack.

If follow-up work tests models with explicit chain-of-thought prompting that surfaces the proof-translation gap as a solvable subproblem, and gaming behavior emerges there but not in standard prompting, that would confirm the constraint is attentional rather than architectural.