Pretraining Recurrent Networks without Recurrence

The deeper implication here is not just faster RNN training but a potential reframing of what 'pretraining' means for recurrent architectures: instead of learning memory dynamics through gradient propagation across time, the model learns them by imitating representations extracted from a Transformer. That makes the Transformer a teacher, not a competitor.

This connects directly to the hardware pressure documented in our coverage of Majestic Labs' Prometheus server (from June 1), which framed the memory wall as a deployment bottleneck for large Transformers. If recurrent models can be pretrained efficiently and then run with lower memory overhead at inference, the hardware argument for Transformer-centric infrastructure weakens at the margin. More broadly, the surgical decoupling logic here rhymes with what SubFit proposed for compression (covered June 1): both papers argue that treating a model as a monolithic training target wastes capacity and that component-aware separation produces better outcomes.

The critical test is whether models pretrained with Supervised Memory Training match BPTT-trained baselines on long-context benchmarks like SCROLLS or RULER, not just short-sequence tasks where the gradient flow advantage of BPTT is minimal. If the gap closes on sequences above 8k tokens, the method is substantive; if it only holds on shorter contexts, the parallelism gains come at a real capability cost.