ORCA-bench tests LLM agents on production incident diagnosis
Researchers have built ORCA-bench, a production-grade evaluation framework that tests whether coding agents can handle real-world incident response. The benchmark simulates authentic on-call scenarios using live instrumented microservices with actual telemetry data from Prometheus, Jaeger, and OpenSearch, then tasks LLM agents with root cause analysis across 1,079 curated scenarios. This work exposes a critical gap between general coding ability and the reasoning demands of production troubleshooting, where agents must synthesize noisy signals across metrics, logs, and traces to diagnose ambiguous failures. The benchmark's SRE-validated ground truth sets a new bar for evaluating agent readiness in infrastructure operations.
Modelwire context
ExplainerORCA-bench's critical insight is that coding ability alone doesn't predict on-call competence. The benchmark isolates a specific reasoning failure mode: agents can write code but struggle to synthesize ambiguous, noisy signals across three independent telemetry systems to reach diagnosis. This is distinct from general coding benchmarks.
This work sits alongside two other evaluation infrastructure papers from this cycle. Change2Task (late July) automates task generation from real code repositories, solving the data sourcing problem. OSReward (same day) addresses whether VLM judges themselves are trustworthy evaluators. ORCA-bench completes a different piece: it validates that the evaluation environment itself (live services, authentic telemetry) matters as much as the task design. Together, these three papers reflect a field-wide recognition that agent benchmarking quality directly constrains how fast and safely the next generation can scale. The difference here is domain specificity: ORCA-bench is SRE-validated, not just algorithmically sound.
If ORCA-bench results correlate with actual on-call performance when agents are deployed against real incidents at participating companies, that validates the benchmark's ecological validity. If the benchmark remains a research artifact with no production adoption within 12 months, it signals that either the scenarios don't generalize or that production teams lack incentive to instrument their systems for agent evaluation.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsORCA-bench · OpenTelemetry · Prometheus · Jaeger · OpenSearch · Grafana
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “ORCA-bench: How Ready Are Language Model Agents for Oncall?”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.