LLM preference judgments fail basic consistency tests

The study doesn't just show LLMs make inconsistent judgments; it quantifies the violation of basic rational axioms (transitivity, equivalence) in cardinal preference outputs. This is distinct from general hallucination or factual errors: it's structural irrationality in the preference signal itself.

This finding sits alongside a cluster of August papers exposing hidden brittleness in LLM-as-oracle systems. The 'Judge, Retrieve, or Abstain' framework from the same week proposes uncertainty quantification to make LLM judges safer; this preference inconsistency paper suggests the problem runs deeper than missing confidence scores. Similarly, the latent-structures study showed that LLMs and humans solve problems via different mechanisms, implying that inferring human utility from LLM outputs may conflate surface agreement with fundamentally misaligned reasoning. And the belief-handling paper revealed that LLM outputs fluctuate based on phrasing rather than stable capability. Together, these suggest that treating LLM outputs as direct proxies for human intent (whether for judging, preferring, or reasoning) systematically underestimates model unreliability.

If preference-learning pipelines that incorporate explicit transitivity constraints (e.g., via constraint satisfaction or post-hoc reconciliation) show measurable improvement in downstream task alignment within the next six months, that confirms inconsistency is a fixable bottleneck. If they don't, it signals the problem is deeper than logical structure and points toward fundamental misalignment between LLM outputs and human utility.