Research·arXiv cs.CL·18h ago

Legal RAG systems fail to retrieve temporally correct law versions

Researchers expose a critical failure mode in legal RAG systems: temporal misgrounding, where models retrieve the current version of a law when an earlier or future version applies. The FiscalQA Pro benchmark, built on 32,436 versioned articles from French tax code spanning 1938-2031, reveals that no evaluated model can recover temporally correct answers in closed-book settings. This work reframes legal question-answering as a temporally-indexed retrieval problem rather than static corpus search, exposing a gap between how production RAG systems operate and what domain-critical applications demand. The finding matters for any regulated industry relying on LLMs for compliance.

Modelwire context

Explainer

The paper's core insight isn't just that models fail on versioned law, but that this failure is invisible to standard RAG metrics. A system can score well on static benchmarks while systematically retrieving legally wrong answers in production because it has no mechanism to track which version of a rule applies to which query date.

This connects directly to two prior findings on evaluation blindness. Like PragMatch exposing how vision-language models rely on shortcut cues rather than genuine reasoning, and like the harmfulness scoring paper showing that internal safety signals don't predict actual jailbreak success, temporal misgrounding reveals a gap between what benchmarks measure and what systems actually do in constrained domains. The KGCaRe work on knowledge graphs also hints at this: symbolic reasoning over structured facts (like versioned rules) outperforms pure retrieval when logical dependencies matter. Legal compliance is the highest-stakes version of that problem.

If major legal tech vendors (LexisNexis, Westlaw, or specialized compliance platforms) announce temporal indexing or version-aware retrieval in their RAG pipelines within the next 12 months, that signals the benchmark moved practice. If they don't, watch whether FiscalQA Pro gets adopted by regulators as a compliance testing requirement for AI systems handling tax or financial law.

Coverage we drew on

KGCaRe: Explainable Complex Conditional Question Answering using Automatic Knowledge Graph Construction and Context Retrieval with LLMs · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsFiscalQA Pro · French tax code

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.