LoRA adapters store far less than parameter counts suggest, placement matters most
Researchers quantified how much information LoRA adapters actually store, finding they compress roughly 2 bits per trainable parameter, far below what parameter counts suggest. Critically, capacity depends less on adapter size than on architectural placement: shifting parameters from attention to MLP layers nearly doubles information density, while removing the frozen base model's structure collapses capacity almost entirely. This challenges the field's assumption that adapters are pure skill vectors, revealing they function partly as compressed data records. The finding reshapes how practitioners should think about adapter design, deployment efficiency, and the implicit memorization risks in parameter-efficient fine-tuning at scale.
Modelwire context
ExplainerThe paper's core finding isn't just that LoRA adapters compress information tightly (2 bits per parameter), but that this capacity floor is almost entirely structural: removing the frozen base model collapses it, while layer placement can nearly double it. This means adapter capacity is not intrinsic to the adapter itself, but borrowed from the architecture it's grafted onto.
This connects directly to the gradient concentration work from earlier this month, which showed that weight importance ranking matters less than we assumed in unlearning pipelines. Both papers challenge the field's habit of treating model components (weights, adapters) as independent agents. The LoRA finding goes further: it suggests that parameter-efficient methods don't actually store skills in isolation, but function as compressed pointers into the base model's learned structure. This has direct implications for the memorization risks flagged here, since adapters may be storing training data not through their own capacity but through their dependency on frozen representations.
If researchers can transfer a LoRA adapter trained on Model A to Model B with the same architecture but different initialization, and capacity drops measurably, that confirms the base model's structure is doing the heavy lifting. Conversely, if adapters trained on different architectures show similar 2-bit compression regardless of layer placement, the structural dependency claim weakens.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsLoRA
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.