How reliable are LLMs when it comes to playing dice?

The more alarming finding isn't the accuracy drop itself but what causes it: models aren't reasoning about probability, they're pattern-matching to familiar problem shapes. When surface framing changes while the underlying math stays identical, scores collapse, which means these models have learned to recognize probability problems rather than solve them.

This connects directly to a cluster of failure-mode research Modelwire has been tracking. The financial LLM audit from early June showed that Bitcoin asset preferences shifted dramatically based on framing context alone, which is structurally the same vulnerability documented here: surface reformulation overrides underlying logic. The eating disorder safety study from the same period found that specific linguistic patterns trigger unsafe outputs, another instance where prompt surface features defeat intended model behavior. Taken together, these papers suggest token bias and framing sensitivity aren't edge cases but recurring architectural properties that show up across domains, from clinical safety to portfolio allocation to basic probability.

Watch whether any of the eight evaluated models releases updated reasoning benchmarks in the next two quarters that specifically include counterintuitive probability variants. If none do, that's evidence the field is still optimizing for canonical problem performance rather than addressing the heuristic shortcut problem this paper identifies.