Frontier models generate correct but wildly inefficient code fixes

The study reveals that SWE-bench's pass-fail scoring masks a second dimension entirely: models producing functionally identical solutions can differ by orders of magnitude in runtime cost. This means current leaderboards conflate correctness with efficiency, treating them as a single binary outcome when they are actually independent variables.

This connects directly to the pattern established in 'Coding-Agent Benchmarks Should Match Their Users' Task Flows' (October 7) and 'TRACE: Diagnosing Verifier Brittleness' (October 8). Those papers showed that static benchmarks miss real-world task diversity and that scoring mechanisms themselves can distort what we measure. This work adds a third layer: even when the verifier is stable and the task is fixed, the metric itself (pass/fail) is incomplete. Production code quality requires both correctness and efficiency, yet SWE-bench optimizes for only the former. The finding echoes the prompt optimization trade-off study from October 9, which showed that maximizing one metric often degrades unscored requirements.

If Claude Opus 5 and GPT 5.6 maintain their relative ranking when benchmarks add latency or memory constraints as co-equal scoring dimensions (rather than post-hoc analysis), this work has shifted evaluation practice. If they don't, and efficiency rankings diverge from correctness rankings, that confirms the current leaderboards are incomplete measures of model utility.