
Recoverable but Not Stationary:Local Linear Structures in Weights and Activations
Researchers challenge the assumption that learned behaviors in neural networks align to fixed linear subspaces. Testing task vectors, LoRA, and activation steering across synthetic and real models, they find that while low-rank task structure exists locally, the basis for recovery drifts significantly over training steps rather than remaining stationary. A new Gaussian local-linear theorem explains why random search succeeds in high dimensions, offering theoretical grounding for parameter-space exploration methods. This work reshapes how practitioners should think about mechanistic control of model behavior: linear directions matter, but their geometry is dynamic, not frozen.62




























