Mamba and Transformers learn nearly identical representations despite architectural differences
Researchers used sparse autoencoders to compare how Mamba state space models and Transformers build internal representations, testing whether architectural constraints force fundamentally different learning strategies. Analyzing 130M-parameter Mamba against 70M Pythia across 10 million tokens, they found striking alignment: 99.98% of learned features clustered together across architectures, suggesting both model families converge on similar conceptual structures despite their opposing design philosophies. This challenges assumptions that information bottlenecks in SSMs produce alien representations, with implications for mechanistic interpretability research and architectural comparisons.
Modelwire context
ExplainerThe paper's real contribution isn't just that SSMs and Transformers learn similar features, but that this similarity persists despite Mamba's information bottleneck (no attention mechanism). This directly falsifies a widespread assumption in mechanistic interpretability: that architectural constraints force qualitatively different internal representations.
This connects to the sparse autoencoder work on representational bias from earlier this month. That research showed how standard dimensionality reduction can inadvertently privilege majority structure in data. Here, the researchers use SAEs as a lens to ask whether architectural differences create hidden representational disparities. The finding that 99.98% of features align suggests the answer is no, at least at this scale. This matters for the broader interpretability agenda: if both architectures converge on the same conceptual space, mechanistic findings from one may transfer to the other, reducing the need to reverse-engineer each family separately.
If this 99.98% alignment holds when tested on Mamba and Transformer models at 1B+ parameters (where architectural differences compound), the result stands. If alignment drops below 95% at larger scales, it signals that feature convergence is a small-model artifact and architectural constraints do matter for learned representations. Watch for follow-up work testing this on models trained on diverse domains beyond language.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsMamba · Transformers · Pythia · Sparse Autoencoders · State Space Models
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.