Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay

The more consequential finding here is not the self-generated replay trick itself, but the hard ceiling it exposes: if a model is already near capacity, no replay strategy rescues it. That reframes capacity as a pre-deployment decision, not a finetuning-time fix.

This connects directly to two threads Modelwire has been tracking. The Prism infrastructure paper from the same day addresses the tooling side of continual instruction tuning, but this work reveals that better tooling cannot compensate for a model that was never given enough headroom to begin with. Separately, the capacity-squeezing work in 'Squeezing Capacity from Multimodal Large Language Models' highlights a parallel tension: researchers are actively trying to extract more from fixed-size models, which, under the findings here, may be pushing those models closer to the saturation threshold where continual adaptation becomes structurally impossible. Together, these papers suggest the field is converging on a capacity planning problem that sits upstream of any algorithmic solution.

Watch whether continual learning benchmarks begin reporting baseline model utilization rates alongside forgetting metrics. If they do not, the saturation constraint this paper identifies will remain invisible in standard comparisons, and practitioners will keep attributing forgetting to method choice rather than model sizing.