Pretraining Recurrent Networks without Recurrence
Researchers propose Supervised Memory Training, a novel pretraining approach that circumvents the sequential bottleneck of backpropagation through time by reformulating RNN training as supervised learning over one-step memory transitions. The method uses a Transformer encoder to extract predictive state representations, then trains the recurrent layer on these labels in parallel. This decoupling addresses two fundamental RNN limitations: computational parallelism during training and gradient flow over long sequences. The work signals a potential shift in how practitioners might pretrain sequence models, particularly relevant as the field balances Transformer dominance with renewed interest in efficient recurrent architectures for inference and streaming applications.62
