Agent optimization gains may not survive repeated cycles, new continual-learning study finds
A new evaluation framework challenges the standard practice of measuring agent optimization as a one-time improvement, instead testing whether gains persist and compound when agents face continual task streams in production. Using Terminal-Bench 2.0, researchers compared three optimization harnesses (GEPA, Meta Harness, RELAI's Verifiable Continua) to determine if repeated optimization cycles degrade prior performance or enable cumulative capability growth. This matters because deployed agents rarely face static benchmarks; they encounter new failures and tasks continuously. The finding that optimizer gains may not compound has direct implications for how teams should architect agent training pipelines and measure real-world robustness.
Modelwire context
ExplainerThe buried implication here is about catastrophic forgetting applied to agent optimizers rather than model weights: even if a harness improves performance on one task batch, subsequent optimization cycles may erode those gains, meaning teams could be running expensive pipelines that net to zero or worse over time.
This is largely disconnected from recent Modelwire coverage. The closest thematic neighbor is the PAT translation work from July 15, which also grapples with LLMs failing to maintain coherence across extended, multi-step contexts, but that paper addresses generation fidelity rather than optimizer stability. The continual-learning framing here belongs to a separate conversation about production agent infrastructure, one that has been building quietly in the evals community but hasn't surfaced prominently in recent coverage on this site.
Watch whether GEPA or RELAI publish follow-up results on Terminal-Bench 2.0 task streams longer than those tested in this paper. If neither does within two quarters, the benchmark will likely remain a research artifact rather than a standard practitioners adopt for pipeline audits.
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsTerminal-Bench 2.0 · GEPA · Meta Harness · RELAI
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.