UsefulBench: Towards Decision-Useful Information as a Target for Information Retrieval

The deeper provocation here is that most IR benchmarks have been measuring the wrong thing entirely. Relevance, the traditional target, asks whether a document relates to a query. Usefulness asks whether retrieving it actually helps someone make a decision or take action, and those two properties diverge more than the field has acknowledged.

This connects directly to the tension surfaced in the DiscoTrace paper from April 16, which found that LLMs systematically favor breadth over human-like selectivity when answering information-seeking questions. Both papers are pointing at the same gap: systems optimized on aggregate similarity metrics miss the judgment calls that make information actionable. The finding that LLM-based retrieval better captures usefulness also rhymes with IG-Search's argument, covered the same day, that step-level information gain is a more meaningful reward signal than trajectory-level relevance. Together these papers suggest a quiet but significant methodological shift is underway in how the field thinks about retrieval quality.

Watch whether established IR benchmarks like BEIR or TREC adopt UsefulBench-style usefulness annotations in their next evaluation rounds. If they do, the relevance-versus-usefulness distinction moves from a research curiosity into standard practice; if they don't, this remains an interesting critique without structural impact.