Transformer attention heads converge to scaled idempotent operators across model scales

The paper isolates a specific geometric property (orientation-driven idempotence in OV operators) as the driver of this behavior, not magnitude or span. This distinction matters because it suggests attention heads aren't just developing specialized functions, but developing them through a particular algebraic constraint that training appears to favor.

This connects directly to the finding from GLM 5.3 Flash that massive models activate only a fraction of their parameters. If roughly 4-8% of attention heads exhibit idempotent closure across models from 2.8B to 235B parameters, it suggests transformers may be developing redundancy or specialization patterns that could be exploited for selective computation. The mechanistic insight here (orientation matters more than magnitude) provides a potential lever for the efficiency optimization that GLM 5.3 Flash demonstrated is possible at scale.

If researchers can identify and prune the non-idempotent heads without degrading downstream task performance, that would validate whether this geometric property correlates with actual redundancy. Watch for follow-up work testing whether removing these heads or constraining their geometry during training produces measurable inference speedups or parameter reductions on standard benchmarks within the next 6 months.