New benchmark tests research agents in real, messy workspaces
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
A new benchmark exposes a critical gap in how research agents are evaluated. Existing benchmarks measure isolated task performance in controlled environments, but real researchers deploy agents into messy, ongoing projects where context matters. ARE-bench shifts evaluation toward stateful workflows, testing whether agents can respect prior decisions and navigate cluttered workspaces with incomplete files and failed attempts. This reframes agent reliability from abstract capability scores to practical utility in actual research pipelines, forcing developers to build systems that integrate with human workflows rather than replace them.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
ARE-bench doesn't just add complexity to existing benchmarks; it measures agent behavior across time and prior decisions within a single workspace. Most prior work (including BioBasicsBench and GDPx) tests isolated task completion. This one tests whether agents can respect constraints they set earlier or inherit from incomplete project history.
This directly extends the pattern established by recent benchmarks like Co-Design Bench (Alibaba, November 8th) and WorkflowArena (November 9th), which both exposed gaps between lab performance and production reality. Those papers showed agents fail on real tasks despite high accuracy on curated ones. ARE-bench goes further by isolating a specific failure mode: agents that perform well on individual subtasks but lose context or ignore prior work when deployed into ongoing projects. The shift mirrors what we saw with the national labs benchmark (November 7th), where 114 of 135 submissions failed integrity checks, suggesting agents struggle with constraints that persist across a session.
If the same frontier models (GPT-6 Astra, Claude Opus 5.5) that scored 26-99 percent on isolated benchmarks score below 40 percent on ARE-bench's stateful tasks, that confirms the gap is real and not an artifact of task selection. If a major agent framework (like AutoGen or LangChain) ships explicit session-state tracking as a result within six months, adoption pressure is genuine.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·The Decoder
Alibaba benchmark reveals 73 percent failure rate for top coding agents on real tasks
Alibaba's Co-Design Bench exposes a critical gap between laboratory performance and production reality for AI coding agents. While models like GPT-6 Astra achieve 99+ percent accuracy on existing benchmarks, they solve only 26 percent of real-world tasks sourced from actual user failures. This 73-point collapse signals that current evaluation frameworks systematically overstate agent reliability, forcing…
MentionsARE-bench
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.