Discover and Prove: An Open-source Agentic Framework for Hard Mode Automated Theorem Proving in Lean 4

The more pointed finding here is methodological: current ATP benchmarks often embed the answer implicitly in how the problem is phrased, meaning models can pattern-match to a solution without actually reasoning through a proof. DAP's 'discover first' design is a direct response to that contamination, not just a performance optimization.

Benchmark integrity has been a recurring concern across recent coverage. The piece on 'Diagnosing LLM Judge Reliability' (arXiv, April 16) surfaced a parallel problem in evaluation: high aggregate scores masking systematic logical failures at the instance level. Both papers are pointing at the same structural issue from different angles — that our measurement tools are quietly flattering the models we use them to assess. The MADE benchmark from the same week made a similar move in medical NLP, introducing a living dataset specifically to fight data contamination. DAP's Hard Mode variants fit that pattern: researchers building harder, cleaner targets because existing ones have been quietly solved by exposure rather than capability.

Watch whether the DAP framework gets adopted by any of the major reasoning model evaluations (DeepMind, OpenAI, or the Lean community's own leaderboards) within the next six months. Adoption there would signal the field accepts the contamination critique; silence would suggest the benchmark ecosystem has too much inertia to self-correct.