LoRA adapters store far less than parameter counts suggest, placement matters most
Source published ·Modelwire updated
Original coverage: arXiv cs.LG ↗·How Modelwire adds context
The development
Researchers quantified how much information LoRA adapters actually store, finding they compress roughly 2 bits per trainable parameter, far below what parameter counts suggest. Critically, capacity depends less on adapter size than on architectural placement: shifting parameters from attention to MLP layers nearly doubles information density, while removing the frozen base model's structure collapses capacity almost entirely. This challenges the field's assumption that adapters are pure skill vectors, revealing they function partly as compressed data records. The finding reshapes how practitioners should think about adapter design, deployment efficiency, and the implicit memorization risks in parameter-efficient fine-tuning at scale.
Modelwire’s AI-generated summary of coverage from arXiv cs.LG.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper's core finding isn't just that LoRA adapters compress information tightly (2 bits per parameter), but that this capacity floor is almost entirely structural: removing the frozen base model collapses it, while layer placement can nearly double it. This means adapter capacity is not intrinsic to the adapter itself, but borrowed from the architecture it's grafted onto.
This connects directly to the gradient concentration work from earlier this month, which showed that weight importance ranking matters less than we assumed in unlearning pipelines. Both papers challenge the field's habit of treating model components (weights, adapters) as independent agents. The LoRA finding goes further: it suggests that parameter-efficient methods don't actually store skills in isolation, but function as compressed pointers into the base model's learned structure. This has direct implications for the memorization risks flagged here, since adapters may be storing training data not through their own capacity but through their dependency on frozen representations.
If researchers can transfer a LoRA adapter trained on Model A to Model B with the same architecture but different initialization, and capacity drops measurably, that confirms the base model's structure is doing the heavy lifting. Conversely, if adapters trained on different architectures show similar 2-bit compression regardless of layer placement, the structural dependency claim weakens.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.LG
Saliency masking may be unnecessary for machine unlearning
A new ablation study challenges a core assumption in machine unlearning: that gradient-based saliency masking is essential for forgetting specific training data while maintaining model performance. Researchers compared SalUn's saliency-selected weight updates against random sparse masks with identical computational budgets on standard vision benchmarks, finding that gradient concentration, not weight importance ranking, drives effective representation-level…
MentionsLoRA
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.