LLM query formats produce inconsistent answers despite identical meaning
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers have identified a fundamental reliability gap in LLM query systems: when the same semantic question is posed in different formats (binary vs. multiple-choice), models produce inconsistent answers. The new PrimaCons benchmark quantifies this cross-primitive inconsistency across Jev model variants, revealing systematic failure patterns that correlate strongly with downstream decision errors (0.91 ROC-AUC). This finding matters because production LLM systems increasingly rely on flexible query primitives for efficiency, yet the paper shows these inconsistencies are widespread. The proposed CPRC calibration method offers a mitigation path, but the core insight signals that semantic equivalence in prompting remains an unsolved problem at scale.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper isolates a failure mode orthogonal to accuracy: models can be numerically correct yet logically inconsistent across equivalent query formulations. This suggests that semantic equivalence in prompting is not a solved property even when individual answers are right.
This connects directly to the frame collapse problem documented in the financial LLMs study from October 28, where models returned numerically correct answers to the wrong question. Both papers expose a gap between apparent competence (right numbers, consistent outputs) and actual reliability (right reasoning, stable semantics). The cross-primitive work also echoes the structured output failures study from October 24, which found that constraint violations and schema breaches are distinct failure modes requiring different fixes. Here, the fix is calibration rather than prompt refinement, but the underlying insight is the same: production LLM systems fail in ways that simple accuracy metrics miss.
If the CPRC calibrator reduces the 0.91 ROC-AUC correlation between cross-primitive inconsistency and downstream errors to below 0.7 when tested on held-out Jev variants not seen during calibration training, that validates the mitigation. If the effect disappears or weakens significantly, it suggests the calibration is overfitting to the benchmark rather than solving the underlying inconsistency.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
LLMs return correct numbers for wrong financial questions
A new study exposes a critical failure mode in financial LLMs: frame collapse, where models return numerically correct answers to the wrong question. Researchers tested this vulnerability using Indian mutual funds as a controlled domain where product variants and tax rules are formally specified. The work reveals that LLMs struggle with underspecified queries by either…
MentionsJev · PrimaCons · Cross-Primitive Residual Calibrator
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Cross-Primitive Consistency in Jev”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.