The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans

The deeper provocation here is methodological: the paper argues that most existing benchmarks inadvertently reward the very pattern-matching they claim to measure against, because benchmark designers and models share the same training-data substrate. Structuring riddles to require literal override of surface associations is an attempt to break that circularity.

This is largely disconnected from the reinforcement learning and causal inference threads running through recent Modelwire coverage. The closest adjacent concern appears in the broader conversation about what benchmark performance actually certifies, a question that sits underneath work like the heavy-ball Q-learning paper from June 25th, where the authors similarly distinguish principled guarantees from empirical intuition. The riddle riddle work is essentially asking the same question one layer up: when a model scores well, does the number mean what we think it means? That skepticism about evaluation validity is a recurring undercurrent across the field, even when the technical domains differ.

The concrete test is whether independent labs can replicate the human-versus-model gap on novel riddle sets they construct themselves, without access to the original stimulus pool. If the gap collapses under independent replication, the effect is likely an artifact of the specific item design rather than a stable probe of reasoning flexibility.