Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers

The deeper provocation here is not just that Dong et al. were wrong in degree, but that the field has been attributing stabilizing work to the wrong architectural components, meaning design intuitions built on that misattribution may have quietly propagated into how practitioners justify MLP layers and depth choices in production models.

This is largely disconnected from the recent Modelwire coverage around semantic bibliometrics and theorem-proving benchmarks. It belongs instead to a quieter but consequential thread in the ML theory space: the effort to build rigorous, falsifiable accounts of why Transformer architectures behave as they do. The theorem-proving benchmarking paper covered around the same date ("Benchmarking Testing in Automated Theorem Proving") is thematically adjacent in one narrow sense, both papers are pushing the field toward more precise, formally grounded evaluation of claims, but the subject matter does not overlap. The rank collapse work sits closer to mechanistic interpretability research and architecture ablation literature, where practitioners need reliable theoretical footing before trusting design heuristics at scale.

Watch whether ablation studies on production models like BERT-base or its successors, run without MLP blocks, show the representational degradation the old consensus predicted. If they do not, this paper's core claim about residual connections doing the stabilizing work holds up empirically, not just measure-theoretically.