Research·arXiv cs.CL·2d ago

Outdated documents flip RAG outputs in 37% of cases across major models

A new failure mode threatens RAG systems at scale: outdated documents actively degrade model performance even when the model would answer correctly without retrieval. Researchers benchmarked 317 knowledge reversals across medicine, law, software, and policy domains, finding that stale evidence flips 30-37% of model outputs from leading open-source systems, rising to 66-75% when models are instructed to prioritize retrieved text. This exposes a critical gap in production RAG pipelines, where temporal validation of sources has lagged behind retrieval speed and scale. The finding reshapes how teams must architect retrieval systems, forcing explicit recency checks and source-date filtering into core workflows rather than treating them as optional safeguards.

Modelwire context

Explainer

The paper isolates a specific failure mode (stale documents flipping correct answers) rather than studying retrieval quality broadly. The key finding is that models instructed to prioritize retrieved text degrade far more sharply (66-75% vs 30-37%), suggesting the problem isn't retrieval alone but how models weight conflicting signals.

This connects directly to 'Highlight-Then-Summarize' from last month, which also identified that naive retrieval doesn't solve long-context reasoning. But where H2S tackled noise filtering through learned compression, this paper surfaces a temporal dimension: the retrieved signal can be actively wrong, not just noisy. The conformal screening work on AI-text detection also shares a deployment concern: both papers identify gaps between benchmark performance and production safety, where systems need explicit verification layers (source-date filtering here, false-positive budgeting there) rather than relying on model behavior alone.

If major RAG vendors (Anthropic, OpenAI, or open-source frameworks like LlamaIndex) ship mandatory source-date filtering as a default rather than optional parameter within the next two quarters, that signals the industry is treating this as a critical control. If they don't, watch whether enterprises building on these platforms begin implementing it independently, which would indicate the burden has shifted downstream.

Coverage we drew on

Highlight-Then-Summarize: Learning to Compress Evidence for Long-Context Understanding · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsLlama · Qwen · RAG

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.