Retrievable Gradients: Continual Post-Training Without Cumulative Weight Drift

The critical detail the summary gestures at but doesn't unpack is the inference-time application: retrieved gradients produce temporary weight shifts that dissolve after the forward pass, meaning the base model parameters never actually change. That's a fundamentally different failure mode than standard fine-tuning, and it's what separates this from prior continual learning work.

This sits in a cluster of approaches all circling the same problem: how do you update what a model knows without retraining it from scratch? DYNA, covered the same day, attacks this through an external temporal knowledge graph that augments frozen model outputs at inference. ReGrad takes a more invasive route by touching the weight space temporarily, which gives it richer parametric behavior than pure retrieval but introduces its own risks around gradient staleness and index maintenance that DYNA's graph-update model largely sidesteps. The two papers together sketch a spectrum: external memory on one end, ephemeral weight perturbation on the other, with neither yet proven at production scale.

The concrete test is whether ReGrad's Gradient Bank remains coherent as document corpora scale past tens of thousands of entries. If retrieval precision degrades at that scale in follow-up benchmarks, the approach collapses into noise rather than targeted adaptation.