The Geometry of Updates: Fisher Alignment at Vocabulary Scale

The key insight is negative: representation-similarity metrics like CKA fail to predict transfer success when models diverge in output heads despite sharing tokenizers. This hidden failure only became visible once computational constraints lifted, revealing a gap between what practitioners measure and what actually determines transfer quality.

This connects directly to the June 25 work on historical Italian tokenization, which decomposed language model failures into measurable, fixable components rather than treating them as monolithic problems. Both papers move beyond treating domain mismatch as a binary barrier and instead offer diagnostic tools for practitioners. The Fisher alignment work extends that logic to the transfer learning stage: instead of brute-force corpus selection or guesswork, you now have a principled metric that accounts for head divergence, not just shared representations. The related work on co-failure ceilings in multi-model systems also shares this DNA: quantifying what ensemble policies cannot overcome, just as this work quantifies what representation similarity cannot predict.

If practitioners building specialized LLM families (SMILES, protein sequences, domain-specific vocabularies) report measurable corpus selection speedups within the next 6 months compared to prior brute-force approaches, the method has cleared the adoption hurdle. Watch whether the authors or downstream work validate the Fisher-kernel approximation on models larger than those in the arXiv version.