Transformers learn Heavy-Ball acceleration beyond supervised gradient descent

The key finding is not that transformers learned an optimization algorithm (prior work showed that with gradient descent), but that trajectory-level supervision, not task-level supervision alone, was the enabling constraint. This suggests the bottleneck in algorithmic discovery isn't model capacity but supervision granularity.

This connects directly to the quantization reasoning work from last week, which identified that distributional misalignment between teacher trajectories and student generation paths causes capability collapse. Here, the inverse problem appears: richer trajectory data lets models converge on better algorithms. The mechanistic interpretability work on representation dynamics from mid-October also becomes relevant: if Heavy-Ball acceleration emerges through training, understanding how attention and layer-wise gating preserve or amplify optimization signals during forward passes would explain why some supervision regimes work and others fail. Together, these papers suggest trajectory fidelity and representation preservation are coupled constraints on what transformers can learn in-context.

If Huang et al. or follow-up work demonstrates that Heavy-Ball trajectories generalize to optimization problems outside the training distribution (different loss landscapes, dimensionalities, or problem classes), that confirms the model learned a genuine algorithmic principle rather than memorizing a specific trajectory pattern. If instead performance degrades sharply on held-out problem families, the discovery is narrower than claimed.