Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics

The paper isolates a specific failure mechanism: LMs don't build unified quantity representations but instead rely on learned associations between individual numbers and unit scales. This explains why performance collapses near decision boundaries, not just why it's imperfect.

This connects directly to the pattern surfaced in recent coverage on multilingual reasoning and domain-specific failure modes. Just as 'Learning When to Translate' revealed that reasoning gaps are often language-specific rather than reasoning-specific, this work shows that quantitative reasoning failures aren't abstract but tied to how models encode particular numerals and units. The eating disorder safety paper and the FRANZ audit framework both exposed how models make systematic errors in high-stakes domains by relying on surface patterns rather than robust internal structure. Quantity comparison sits in that same category: a task where surface heuristics feel sufficient until precision matters.

If researchers can show that explicit unit normalization during training (converting all quantities to a canonical reference frame before the model sees them) restores performance near decision boundaries, that confirms the diagnosis. If performance remains degraded even with normalized inputs, the problem runs deeper than representation and points to architectural limits in how transformers handle numerical reasoning.