LLMs fail legal deadline math despite correct arithmetic, hybrid approach succeeds
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers demonstrate a critical failure mode in LLM reasoning: language models confidently produce wrong answers on legal deadline calculations despite getting the arithmetic correct. By extracting temporal dependencies into a structured graph and delegating computation to a deterministic calendar engine, the team achieved near-perfect accuracy on UK Employment Appeal Tribunal cases where the strongest LLM baseline failed in 30% of instances. This work signals a broader architectural shift in AI systems handling high-stakes domains: hybrid approaches that separate symbolic reasoning from neural pattern-matching may be essential where errors carry legal consequences.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper's core insight isn't that LLMs fail at deadline math, but that they fail specifically when temporal reasoning requires extracting implicit dependencies from unstructured text before computation. The hybrid approach delegates the extraction to a graph structure, not the neural component.
This connects directly to two August findings on temporal reasoning gaps. The earlier work on whether LLMs encode the current year identified fragile temporal grounding and task-dependent mechanisms. This paper goes further by showing that even when models get the arithmetic right, they misidentify which dates matter and in what sequence. The option-bias paper from the same week also exposed how legal benchmarks can hide reasoning failures behind surface-level correctness, which mirrors the 30% failure rate here despite confident outputs.
If the same temporal dependency extraction approach improves performance on the Ukrainian judicial exam benchmark mentioned in recent coverage (the one with 11,990 items before filtering), that would validate whether this is a general legal reasoning pattern or specific to UK employment law. Watch for follow-up work applying this to non-legal domains with implicit temporal constraints, like narrative understanding or historical reasoning.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Language models show inconsistent temporal grounding across task types
Researchers have identified a fundamental gap in how language models represent temporal context. By testing both implicit and explicit probing tasks, they found that instruction-tuned models anchor to their training cutoff with surprising precision, yet base models reveal divergent internal mechanisms between factual recall and direct knowledge queries. This work exposes a critical blind spot…
MentionsUK Employment Appeal Tribunal · temporal dependency graphs · language models
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Time as Structure: Temporal Dependency Graphs for Verifiable Deadline Computation over Legal Documents”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.