SWE-Bench ProMax resets code agent evaluation with refactoring focus
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Existing code-generation benchmarks are collapsing under their own success. A new audit reveals that 60% of SWE-bench Verified tasks contain broken test cases, while frontier models exploit training-data leakage to artificially inflate scores. SWE-Bench ProMax addresses this crisis by shifting focus to multilingual code refactoring, a task class that demands coordinated, semantics-preserving edits across multiple files. This pivot matters because it forces agents to demonstrate genuine reasoning rather than pattern matching, resetting the evaluation bar for software engineering AI and exposing which systems have actually learned to code versus which have merely memorized.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
Skeptical readOur AI-generated reading of the wider context and the next developments to watch.
SWE-Bench ProMax doesn't address why the original benchmark failed (training data leakage, broken test cases); it sidesteps the problem by moving to a new task domain. That's a valid triage move, but it doesn't explain whether multilingual refactoring will resist the same contamination pressures that collapsed the prior benchmark, or if this is just buying time before the next audit finds similar rot.
This connects directly to the broader evaluation crisis surfaced in recent work on diagnostic stress testing (Decoding-Level Taboo, August) and token-level metric collapse (Mismatch Matters, same period). Those papers exposed how benchmarks measure performance under ideal conditions while missing fragility in real constraints. SWE-Bench ProMax is attempting to tighten the constraint set by forcing semantics-preserving edits across files, but it's still a benchmark-level fix, not a structural fix to how we validate that models have learned versus memorized. The Dutch municipal LLM evaluation framework from August offers a contrasting approach: instead of pivoting to harder tasks, it operationalizes multiple evaluation dimensions (factuality, bias, transparency) to catch different failure modes. SWE-Bench ProMax assumes one harder task will reveal truth; the Dutch work assumes truth requires multiple lenses.
If SWE-Bench ProMax scores remain stable across three consecutive model releases without evidence of training data leakage (check for exact task overlap in model cards and training documentation), the pivot worked. If a new audit within 12 months finds similar contamination or broken test cases in ProMax itself, the field has confirmed that task-switching is a temporary patch, not a solution.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Stress test reveals LLM performance collapse outside narrow optimization paths
Researchers have developed Decoding-Level Taboo, a runtime diagnostic that exposes gaps between LLM benchmark performance and real-world robustness by forcing models into unfamiliar generation paths. The method masks high-probability tokens during inference, compelling models to generate circumlocutions and revealing how quickly performance degrades when models deviate from their narrow optimization corridors. This stress test addresses…
MentionsSWE-Bench · SWE-Bench ProMax · SWE-Bench Verified
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.