Efficient Pre-Training with Token Superposition

The paper's real claim isn't a new architecture but a drop-in efficiency layer that works without touching parallelism or model structure, which means adoption friction is unusually low compared to most pre-training optimizations. That low-friction profile is what makes the economics argument credible rather than theoretical.

This lands in a cluster of concurrent work attacking pre-training costs from different angles. The Lighthouse Attention paper from the same day targets sequence-length scaling through hierarchical compression, while the Randomized Subspace Nesterov paper from early May addresses gradient computation efficiency in distributed settings. TST is complementary to both: it operates at the token-batching level rather than the attention or optimizer level, meaning a lab could stack all three without obvious conflicts. The MIT superposition study from May 3rd adds an interesting wrinkle, since that work identifies superposition as the mechanistic driver of scaling laws, and TST's multi-hot objective deliberately induces a form of token superposition during training. Whether that mechanistic overlap is coincidental or whether TST inadvertently exploits the same representational dynamics MIT described is an open question the paper does not appear to address.

If a frontier lab cites TST in a training infrastructure post or technical report within the next two quarters, that's a signal the drop-in adoption story holds under production conditions. Silence from that tier by end of 2026 would suggest the gains don't survive the messy realities of large-scale distributed runs.