New benchmark exposes how badly AI struggles with real knowledge work

The 3 percent figure matters less as a headline shock and more as a methodological signal: most existing benchmarks measure isolated, well-scoped tasks, and this one appears to test chained, context-dependent workflows where errors compound across steps rather than reset between questions.

This is largely disconnected from recent activity in our archive, as we have no prior coverage to anchor it to. It belongs to a longer-running conversation in the AI evaluation space about whether current benchmarks actually predict deployment value. That conversation has been building across academic venues and enterprise AI teams for roughly two years, driven by repeated cases where models that score well on standardized tests fail on the messier, ambiguous inputs that real professional work generates. The benchmark described here appears to be a direct response to that criticism, attempting to operationalize 'realistic' in a way that prior evals have avoided.

Watch whether the benchmark's authors release a public leaderboard with third-party model submissions within the next 60 days. If major labs engage with it formally rather than ignoring it, that signals the methodology has enough credibility to influence internal roadmaps.