Test-time training shrinks LLM cache overhead without retraining
Source published ·Modelwire updated
Original coverage: arXiv cs.LG ↗·How Modelwire adds context
The development
Researchers propose Latent Compaction, a test-time training method that shrinks key-value caches in language models without sacrificing context quality. Rather than compressing caches upfront, the technique learns document-specific compact representations through lightweight optimization using only generic prompts, keeping model weights frozen. This addresses a critical bottleneck in long-context inference: memory overhead and computational cost during generation. The approach combines reconstruction importance, teacher attention signals, and diversity heuristics to initialize cache entries, then refines them via distillation from the full cache. For practitioners deploying LLMs at scale, this offers a practical path to cheaper inference on lengthy documents without retraining.
Modelwire’s AI-generated summary of coverage from arXiv cs.LG.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
Latent Compaction defers compression decisions until inference time, learning which cache entries matter for each specific document rather than applying a fixed compression strategy upfront. This is distinct from layer-wise symmetry exploitation (VFold) or token merging (zip2zip++) because it treats compression as a per-input optimization problem, not a model-level or architecture-level one.
This extends the inference-time optimization wave documented in recent coverage. Like zip2zip++ (October 9), it avoids retraining and recovers quality through architectural refinement rather than brute-force pruning. But where zip2zip++ compresses tokens, Latent Compaction compresses the cache itself, addressing the same bottleneck that VFold tackled (October 8) via layer symmetries. The key difference: VFold is static (applies the same compression across all inputs), while this method adapts per document. The Attic-KV finding (October 8) that rehearsal under tight budgets hurts performance is relevant here too, since Latent Compaction's lightweight optimization avoids the full-context rereading trap by using only generic prompts.
If Latent Compaction maintains quality at 10-20% cache size on the RULER benchmark (the standard for long-context eval), and if that holds on both synthetic and natural-text tasks without task-specific tuning, the per-input optimization angle is real. If gains collapse when tested on out-of-distribution documents or require prompt engineering to work, the method is overfitting to the evaluation setup.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Rehearsing context hurts KV cache compression at tight budgets
Attic-KV challenges a core assumption in KV cache compression: that rehearsing (rereading) context before pruning improves retention. The paper demonstrates the opposite under tight memory budgets. At 3% compression, full-context rehearsal drops performance to 31.5/96.5 on RULER and underperforms no-rehearsal baselines on natural-text tasks. The insight is mechanically simple but consequential: spreading a fixed budget…
MentionsLatent Compaction
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Latent Compaction with Test-Time Training”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.