Aligning Dense Retrievers with LLM Utility via DistillationAligning Dense Retrievers with LLM Utility via Distillation

The core insight here is architectural: current RAG pipelines treat retrieval and generation as loosely coupled stages, scoring documents by semantic similarity to a query rather than by how much they actually help a downstream model produce a correct answer. This work argues that gap is trainable away, by distilling LLM utility signals into the embedding space itself during retrieval model training.

This is largely disconnected from the two related stories currently on Modelwire, which cover scaling law fitting costs and neural network surrogates for optimization problems. The relevant context sits elsewhere: RAG has become the dominant architecture for grounding LLM outputs in external knowledge, and the bottleneck has quietly shifted from generation quality to retrieval precision. The practical pressure is real because re-ranking with a large model at inference time is expensive at scale, and this paper directly targets that cost by front-loading the alignment work into the retriever.

The critical test is whether Utility-Modulated InfoNCE trained retrievers hold their accuracy advantage on retrieval benchmarks that use held-out LLMs for utility scoring, not the same model family used during distillation. If performance degrades significantly across model families, the method may be fitting to a specific LLM's preferences rather than capturing generalizable document utility.