Surprisal theory lacks falsifiability without constraining language models
A new paper challenges the empirical foundation of surprisal theory, a dominant framework linking language model predictions to human reading difficulty. The author demonstrates that any observed difficulty pattern can be fit by some language model, rendering the theory unfalsifiable without additional constraints. This exposes a two-decade implicit assumption in psycholinguistics: that the relevant model is the corpus distribution. The finding threatens a core justification for using LLM loss as a cognitive proxy and forces the field to either specify what makes a language model theoretically valid for human cognition or abandon surprisal as a predictive framework.
Modelwire context
ExplainerThe paper's real contribution isn't just that surprisal theory can be fit by any model. It's the identification of an unstated premise: that psycholinguists have been implicitly assuming the 'right' language model is one trained on natural text corpora, without ever justifying that choice theoretically.
This is largely disconnected from recent activity in the space of LLM capability launches and benchmarking. Instead it belongs to a slower-moving conversation about whether language model internals are valid proxies for human cognition at all. The finding matters because it pulls back the curtain on a two-decade empirical program in psycholinguistics that has borrowed credibility from LLMs without establishing the theoretical link. If the field can't specify what makes a model 'valid' for cognition, then LLM loss becomes just another curve-fitting tool, not a window into the mind.
Watch whether major psycholinguistics labs (MIT, UC San Diego, Edinburgh) publish follow-ups within the next 12 months that either propose explicit constraints on which language models count as cognitively plausible, or pivot to testing surprisal against non-LLM baselines. If silence prevails, the field has tacitly accepted the critique.
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsSurprisal theory · Language models · Psycholinguistics
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Surprisal Theory is Tautological (without Rational Grounding)”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.