LLMs fail legal deadline math despite correct arithmetic, hybrid approach succeeds
Researchers demonstrate a critical failure mode in LLM reasoning: language models confidently produce wrong answers on legal deadline calculations despite getting the arithmetic correct. By extracting temporal dependencies into a structured graph and delegating computation to a deterministic calendar engine, the team achieved near-perfect accuracy on UK Employment Appeal Tribunal cases where the strongest LLM baseline failed in 30% of instances. This work signals a broader architectural shift in AI systems handling high-stakes domains: hybrid approaches that separate symbolic reasoning from neural pattern-matching may be essential where errors carry legal consequences.
Modelwire context
ExplainerThe paper's core insight isn't that LLMs fail at deadline math, but that they fail specifically when temporal reasoning requires extracting implicit dependencies from unstructured text before computation. The hybrid approach delegates the extraction to a graph structure, not the neural component.
This connects directly to two August findings on temporal reasoning gaps. The earlier work on whether LLMs encode the current year identified fragile temporal grounding and task-dependent mechanisms. This paper goes further by showing that even when models get the arithmetic right, they misidentify which dates matter and in what sequence. The option-bias paper from the same week also exposed how legal benchmarks can hide reasoning failures behind surface-level correctness, which mirrors the 30% failure rate here despite confident outputs.
If the same temporal dependency extraction approach improves performance on the Ukrainian judicial exam benchmark mentioned in recent coverage (the one with 11,990 items before filtering), that would validate whether this is a general legal reasoning pattern or specific to UK employment law. Watch for follow-up work applying this to non-legal domains with implicit temporal constraints, like narrative understanding or historical reasoning.
Coverage we drew on
- Do Language Models Consistently Encode the Current Year? · arXiv cs.CL
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsUK Employment Appeal Tribunal · temporal dependency graphs · language models
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Time as Structure: Temporal Dependency Graphs for Verifiable Deadline Computation over Legal Documents”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.