SSMs and transformers organize learned knowledge through opposite geometric strategies

The paper doesn't just document that SSMs and transformers organize information differently; it quantifies the geometric divergence (uniform vs. concentrated) and shows this gap widens with depth. The implication is that performance parity between the two architectures masks fundamentally different computational strategies, not convergence toward a shared solution.

This connects directly to the C-RASP work on length generalization from earlier today. Both papers are asking how transformers actually use their representational capacity to solve language tasks. Where C-RASP+ provides computable bounds on what transformers can generalize to, this geometry study reveals the structural substrate those bounds operate on. If transformers concentrate knowledge into a single dominant direction, that concentration may be what enables or constrains length generalization in the first place. The two findings together suggest that future architecture choices (hybrid SSM-transformer designs, or pure SSM scaling) should be informed by whether the task requires the geometric properties of one approach or the other.

If Mamba-scale models trained on identical data and compute budgets begin to diverge from transformer performance on out-of-distribution length generalization tasks within the next six months, that would confirm the geometric difference has practical consequences beyond in-distribution benchmarks. If they remain matched, the finding stays mechanistically interesting but architecturally neutral.