Convergent Evolution: How Different Language Models Learn Similar Number Representations

The buried implication here is diagnostic: convergence on periodic representations is apparently necessary but not sufficient for useful numerical reasoning. A model can learn the right shape of number representation and still fail at modular arithmetic if the geometry doesn't separate cleanly, which means probing for Fourier structure alone won't tell you whether a model can actually do the math.

This connects most directly to the April 16 piece on 'How Embeddings Shape Graph Neural Networks,' which similarly isolated the embedding layer as a variable independent of backbone architecture. Both papers are asking the same underlying question: how much does the representation format determine downstream capability, versus the training regime around it? The convergent evolution framing here adds a wrinkle that the GNN paper didn't address, namely that architectures can arrive at structurally similar representations through entirely different paths, yet still diverge on what those representations support. The optimizer benchmarking piece from April 16 is also quietly relevant, since this paper flags optimizer choice as one of the factors determining whether geometric separability is achieved.

Watch whether any of the four architecture families studied here shows consistent failure on geometric separability across multiple tokenizer configurations. If tokenizer choice reliably breaks separability even when architecture and optimizer are held constant, that shifts the practical intervention point from training to preprocessing.