New benchmark measures how well AI retrieves papers for scientific ideation

The benchmark's real novelty isn't just scale or coverage, but the claim that existing retrieval metrics miss a structural feature of scientific reasoning: that researchers need different kinds of results at different abstraction levels, and systems should be scored on whether they can distinguish and surface all three.

This connects directly to the pattern visible in recent benchmarking work like MCR-Bench and SWE-Prime, both released this month. Those papers exposed gaps in how we evaluate systems by moving beyond static, single-pass metrics toward capturing the actual workflows researchers and developers navigate. RATIO applies the same logic to literature discovery: the gap isn't in retrieval speed or recall, but in whether systems understand that a query for 'how to implement attention' needs different answers than 'what is attention theoretically' or 'has anyone solved this exact problem.' The work signals that evaluation frameworks are shifting from binary relevance toward task-aligned reasoning support.

If downstream work shows that systems trained to optimize for RATIO's three operations outperform standard retrieval baselines on real researcher workflows (measured via user studies or agent task completion), the framework moves from conceptual to predictive. If adoption stalls at the benchmark level without integration into production retrieval systems, it remains a useful taxonomy without practical traction.