Variance Is Not Importance: Structural Analysis of Transformer Compressibility Across Model Scales

The practical implication buried in this finding is that many deployed pruning and quantization pipelines are likely discarding the wrong components, because they rely on activation variance as a cheap heuristic for identifying what the model 'cares about.' The linearity caveat about input distributions is equally important: it means compression benchmarks run on clean or narrow datasets may not predict real-world degradation.

This connects directly to the K-Token Merging paper covered on April 16, which proposed compressing token sequences in latent space under the implicit assumption that certain representational structure is safely collapsible. That paper's LoRA-adapted inference pipeline would be directly affected if the components it treats as low-importance are misidentified by variance-based signals. More broadly, the AdaSplash-2 work from the same week, which achieves sparsity through input-dependent attention, at least acknowledges input distribution sensitivity in a way that this new paper suggests is the right instinct. The field is converging on a problem: compression techniques are outpacing the theoretical tools needed to verify what they're actually removing.

Watch whether follow-up work applies these structural diagnostics to Mistral-class models under instruction-tuning distributions specifically, since that is the gap between the paper's controlled conditions and the settings where compression decisions are actually made in production.