Closed-loop revision shows wide model gaps despite perfect feedback
Researchers benchmarked how well large language models actually respond to corrective feedback in closed-loop revision workflows. Using a controlled protocol with deterministic verifiers that report all constraint violations, they tested 19 models and found success rates ranging from 17.4% to 99.8%, with persistent gaps even under identical conditions. The findings suggest that model-specific revision behavior is reproducible and largely resistant to scaling or post-training improvements. This matters for production systems relying on iterative refinement: exact feedback alone doesn't guarantee control, and architectural or training differences create hard ceilings on revision capability that current techniques don't overcome.
Modelwire context
ExplainerThe paper's core finding is that feedback precision doesn't translate to revision success, and that model-specific ceilings appear immutable to standard scaling or training approaches. This inverts a common assumption in iterative refinement workflows: more detailed error reports alone won't fix the underlying architectural or training differences that determine whether a model can actually follow corrections.
This connects directly to the SWE-Flux benchmark from earlier this week, which exposed gaps in LLMs' ability to reason about runtime behavior rather than static patterns. Both papers reveal that capability limits aren't simply about data quality or prompt precision, but about fundamental model properties that resist conventional fixes. The revision ceiling here mirrors the dynamic reasoning gap there: you can give the model perfect information, but if its architecture or training didn't equip it to use that information, performance plateaus. The PASTABench work on real-time agent safety also hinges on this same problem: if models can't reliably respond to corrective signals mid-trajectory, intervention frameworks become less reliable than they appear.
If Claude or GPT-4 show revision success rates above 90% on this benchmark while Llama variants stay below 70% despite identical feedback protocols, that confirms the finding is architectural rather than training-data dependent. Watch whether any vendor publishes ablations showing whether fine-tuning on revision tasks narrows these gaps within the next two quarters; if not, it suggests the ceiling is baked into model design.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsGPT · Claude · Llama · LLMs
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Exact Feedback Is Not Control: Evaluating Text-based Closed-Loop Revision in LLMs”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.