Alibaba benchmark reveals 73 percent failure rate for top coding agents on real tasks
Source published ·Modelwire updated
Original coverage: The Decoder ↗·How Modelwire adds context
The development
Alibaba's Co-Design Bench exposes a critical gap between laboratory performance and production reality for AI coding agents. While models like GPT-6 Astra achieve 99+ percent accuracy on existing benchmarks, they solve only 26 percent of real-world tasks sourced from actual user failures. This 73-point collapse signals that current evaluation frameworks systematically overstate agent reliability, forcing enterprises to budget for substantial human review overhead. The finding reshapes expectations around autonomous code generation and highlights why benchmark diversity matters for credible capability assessment.
Modelwire’s AI-generated summary of coverage from The Decoder.
Modelwire analysis
Analyst takeOur AI-generated reading of the wider context and the next developments to watch.
The 73-point collapse between benchmark and production performance isn't just a measurement curiosity. It has a direct cost implication: any enterprise that staffed or priced an agentic coding workflow against published benchmark numbers is now looking at a labor model that is roughly four times more human-intensive than the headline figures suggested.
This is the third data point in four days establishing the same pattern across different domains. The SemiAnalysis GDPx benchmark (November 5) showed no agent above 12 percent on professional knowledge work, and the national labs research agent test (November 7) found 114 of 135 submissions failed integrity checks entirely. Co-Design Bench adds coding to that list, which matters because coding has been the most commercially mature agentic use case and the one where vendor confidence has been highest. Taken together, these results suggest the benchmark collapse is not domain-specific but structural: evaluations built on curated, solvable tasks systematically exclude the failure modes that define real deployment.
Watch whether OpenAI or any major coding agent vendor publishes a Co-Design Bench score within the next 60 days. Silence on a public benchmark with this visibility is itself a signal about how those numbers would land.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·The Decoder
SemiAnalysis GDPx benchmark shows AI agents plateau below 12 percent on knowledge work
SemiAnalysis has released GDPx, a benchmark designed to measure AI agent performance on realistic knowledge work tasks spanning chip design, quantitative trading, and related domains. The results reveal a significant capability gap: the highest-performing models, Claude Opus 5.5 and Gemini 4 Carbon, achieved only 12 percent accuracy. This finding underscores that despite recent advances in…
MentionsAlibaba · Co-Design Bench · GPT-6 Astra · OpenAI
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. The Decoder originally reported this story as “AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test”. The full content lives on the-decoder.com. If you’re a publisher and want a different summarization policy for your work, see our takedown page.