Reasoning models fail on task variants despite solving originals
Researchers have exposed a critical gap in how reasoning models generalize: they often fail on structurally equivalent task variants despite solving the original problem. By adapting cognitive science rule induction tasks and creating isomorphic variations through recombination and substitution, the study reveals that current models lack systematicity, the cognitive principle that understanding one concept should transfer to close variations. This finding challenges assumptions about reasoning model robustness and suggests their problem-solving may be brittle and surface-level rather than deeply compositional, with implications for deployment in domains requiring reliable generalization.
Modelwire context
ExplainerThe study doesn't just show models fail on harder problems; it shows they fail on structurally identical problems with different surface features. This suggests models memorize task patterns rather than extract underlying rules, a distinction that reframes what 'reasoning' means in current systems.
This connects directly to the curriculum learning work from yesterday (Teacher-Guided Curriculum for Data-Efficient RLVR), which assumes models can learn generalizable problem-solving strategies through scaffolding. If models lack systematicity, curriculum approaches may only teach surface-level pattern matching more efficiently, not deeper compositional reasoning. The finding also echoes the tool availability study from the same day, which found models abandon reasoning when tools are present; both suggest reasoning behavior is brittle and context-dependent rather than robust. The gap between solving a problem and solving its variants is precisely the kind of failure mode that would compound in multi-step reasoning systems like GraMRAG, where persistent memory can't compensate for weak compositional foundations.
If the same models tested here show improved systematicity after training on the curriculum approach from the related work, that would suggest the brittleness is remediable through data strategy. If they don't, it signals a harder architectural problem that no amount of training data alone will fix.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Thought without systematicity? Evaluating reasoning models on rule induction tasks”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.