Looped transformers unlock hidden multi-step reasoning through intermediate token passing
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers demonstrate that standard transformer architectures fail to perform multi-step reasoning when intermediate entities don't appear in training data, revealing a fundamental gap between next-token prediction and compositional reasoning. The Baton family of looped transformers addresses this by explicitly passing intermediate representations between reasoning steps, achieving near-perfect performance on synthetic two-hop questions. This work exposes how current training objectives may obscure models' capacity for systematic reasoning and suggests architectural modifications could unlock latent compositional abilities, with implications for how we design and evaluate reasoning in language models.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper reveals that transformers trained on next-token prediction can possess compositional reasoning ability that remains latent and invisible under standard evaluation. The gap isn't capability but observability: models fail not because they can't reason, but because the training objective never forces them to externalize intermediate steps.
This directly echoes the phase diagram work from mid-November, which showed how representation geometry evolves through layers in ways that don't map cleanly to observable outputs. Both papers suggest that what we measure in language models (token-by-token accuracy) may systematically underestimate what's actually happening in the weights. The Baton architecture's explicit intermediate passing also parallels the Latent Bridge technique from the same week, which decouples reasoning from generation by making internal representations explicit and transferable between models. Together, these three papers form a coherent argument: current inference-time behavior is constrained not by model capacity but by architectural design choices that hide or suppress intermediate reasoning.
If the Baton family shows comparable gains on held-out multi-hop benchmarks (QASC, HotpotQA) where intermediate entities DO appear in training data, that confirms the mechanism is about training signal, not just architectural novelty. If performance collapses on those benchmarks, the looped design may be solving a narrower problem than the paper suggests.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Phase diagram reveals how transformer geometry shapes inference behavior
Researchers have mapped how representation geometry evolves through transformer layers during inference, using two metrics: document distinguishability and document dimension. Their phase diagram reveals a critical insight: untrained models naturally drift toward collapse or noise regions depending on initialization scale, while trained models follow different trajectories. This work bridges the gap between internal transformer mechanics…
MentionsBaton · DiscoLoop · transformer
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Latent Fact Composition under Next-Token Training”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.