Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers have identified a critical vulnerability in LLM-based legal systems: models fail when statutory law evolves beyond their training data, either by applying outdated rules or over-weighting recent provisions regardless of temporal relevance. A new benchmark of 312 German statutory QA pairs tests how GPT, Claude, and DeepSeek handle temporal reasoning across vanilla, web-search, and retrieval-augmented inference modes. This work exposes a fundamental mismatch between static parametric knowledge and dynamic legal systems, forcing practitioners to rethink deployment strategies for high-stakes domains where legal accuracy depends on knowing which version of a rule applies to a given fact pattern.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper's sharpest finding isn't just that models get outdated law wrong, it's that they also fail in the opposite direction, over-weighting recent statutory text even when an older version governs the facts at hand. That bidirectional failure means retrieval augmentation alone can't fix the problem without a layer of temporal reasoning about which version is legally operative for a given date.
The compliance and high-stakes data angle connects loosely to the Structure-Guided Entity Resolution work published the same day, which flagged a parallel theme: general LLM capability breaks down when domain constraints are rigid and errors are consequential. Both papers are essentially arguing that production deployment in regulated contexts requires more than prompting a capable base model. Where SGER addressed structural ambiguity in identity data, this paper addresses temporal ambiguity in legal corpora. Neither problem yields to off-the-shelf RAG pipelines without domain-specific scaffolding.
Watch whether any of the three tested model families, GPT, Claude, or DeepSeek, release legal-domain system cards or retrieval configurations that explicitly address version-dated statutory lookup. If none do within the next two quarters, this benchmark will likely be adopted by third-party legal AI auditors before the labs respond directly.
This interpretation is generated from the summary above and available source metadata. Our methodology · Report an error
MentionsOpenAI · Anthropic · DeepSeek · GPT · Claude
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.