Legal LLMs cite the right laws but don't follow them
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
A new audit reveals a critical gap in how large language models justify legal reasoning. Researchers tested whether models actually ground their verdicts in the authorities they cite, by swapping named statutes and precedents for unrelated ones while holding facts constant. Results show models reliably name the correct legal source 67-100% of the time, yet their actual decisions shift based on the substituted authority only 0-77% of the time across benchmarks. This disconnect exposes that citation in legal AI may be performative rather than causal, raising serious questions about whether models truly reason from law or merely pattern-match plausible-sounding justifications. The finding matters for deployment in high-stakes domains where explainability is assumed to equal trustworthiness.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The audit doesn't just measure whether models cite correctly; it isolates whether citation causally drives the verdict by replacing authorities and watching if decisions change. This counterfactual approach is sharper than asking whether models can explain their reasoning after the fact.
This connects directly to the meta-faithfulness problem exposed in the SIFT paper from early October. That work showed chain-of-thought detectors themselves fail under distribution shift, making it impossible to trust that a reasoning trace is actually faithful. This legal audit takes a different angle: even when a model names the right source, the counterfactual test reveals the citation may be epiphenomenal. Together, these papers suggest the field has been conflating explainability with causality. The earlier work on reasoning-token spikes and context-sensitive faithfulness benchmarks also point to the same gap: we've been measuring the wrong thing. This legal domain test makes that failure concrete and measurable.
If the same counterfactual substitution method applied to general-domain benchmarks (like those used in SIFT or PFaithBench) shows similar 0-77% decision shifts, that confirms this is a fundamental property of how LLMs ground reasoning, not a legal-domain artifact. If results hold only in law, it suggests legal reasoning may be especially vulnerable to pattern-matching over principled inference.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.LG
Reasoning audits fail under distribution shift, new research shows
Researchers formalize meta-faithfulness, a critical gap in how we validate reasoning audits themselves. The work proves that chain-of-thought detectors fail under distribution shift in ways independent of their training accuracy, and that no detector can distinguish truly faithful reasoning from epiphenomenal patterns using only intervention signatures. This exposes a foundational weakness in current model auditing…
MentionsCaseHOLD · ECHR · SCOTUS · ContractNLI
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.