Claude Code leads on speed but trails on cost in agent framework benchmark

The benchmark doesn't just measure speed; it exposes that framework choice is now decoupled from capability parity. All four frameworks hit comparable success rates, meaning the decision tree for teams has collapsed into a single variable: cost per task.

This connects directly to the inference optimization work from Baseten (early August) showing that speed and cost efficiency have become primary differentiators. But where that piece focused on model-level optimization, this benchmark reveals the same pressure is now reshaping the agent framework layer. The research software modernization story from OpenAI also matters here: if agents can't be trusted without human validation anyway, paying 2.7x for speed becomes harder to justify unless latency directly blocks a workflow. The reliability work on prompt compression and red teaming suggests teams are already thinking about operational risk and control costs, not just raw performance.

If Composio releases a follow-up benchmark comparing cost-per-successful-task across different model backends (Claude 3.5 Sonnet vs Deepseek vs open models), that signals whether the 2.7x premium is Claude-specific or framework-specific. If OpenCode or a competitor launches a production case study showing sub-$0.10 per-task economics at scale within the next two quarters, the cost gap becomes a real adoption lever.