Evaluating Commercial AI Chatbots as News Intermediaries

The real finding isn't the accuracy numbers themselves but the mechanism: these chatbots likely perform well on constrained tests partly because their search and retrieval layers do heavy lifting that disappears when the scaffolding is removed. That means published benchmark scores for news tasks may be measuring the pipeline, not the model.

This is largely disconnected from recent activity in our archive, as we have no prior coverage to anchor it to. It belongs to a growing body of work questioning whether AI evaluation methodology keeps pace with deployment realities. The broader context is that as GPT-5, Claude 4.5, and Gemini 3 have all been positioned partly as research and information tools, the question of how they handle live, multilingual, time-sensitive content becomes commercially significant, not just academically interesting.

Watch whether any of the six vendors respond by publishing their own retrieval-ablated benchmarks on news corpora. If none do within six months of this paper, that silence is informative about how much of their accuracy depends on infrastructure they prefer not to isolate.