EncBank cuts LLM query memory overhead by 72 percent via layer freezing
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
EncBank addresses a persistent efficiency gap in LLM inference: redundant encoding of shared documents across multiple queries. By freezing a pretrained model's lower layers as a fixed encoder and caching their outputs at reduced precision, the approach cuts persistent GPU memory by 72% while maintaining near-native performance across benchmarks. The technique uses a shared suffix adapter that works across quantization levels without retraining, making it practical for production systems handling repeated document queries. This matters for cost-sensitive deployments where document-heavy workloads dominate, particularly in RAG and multi-turn retrieval scenarios.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
EncBank's core insight is orthogonal to KV cache optimization: it targets the encoder itself, not the attention cache. By freezing lower layers and caching their outputs at reduced precision across queries, it sidesteps recomputation entirely rather than compressing what's already computed. This is a different lever than the quantization and eviction strategies dominating recent work.
The last five days have seen a flood of KV cache techniques (TaSQ on 1-bit compression, Page-EntroKV on grouped-query eviction, ResidualQuant on looped models, Hybrid Latent Attention on recurrent architectures, and KV² on multi-query prefill). EncBank doesn't compete with these; it addresses the layer before them. Where KV² tackles redundant scoring across queries on the same prefilled context, EncBank eliminates redundant encoding of the context itself. The two could stack. This also echoes the in-parameter memory survey from October 6, which flagged the cost of repeated encoding as a bottleneck that post-training techniques might sidestep more elegantly than context-level tricks.
If Qwen or other vendors ship EncBank as a production option in their serving frameworks (vLLM, SGLang) within the next two quarters and report real-world memory savings on RAG workloads that match the 72% claim, that confirms the technique generalizes beyond the paper's benchmarks. If adoption stalls because the suffix adapter requires per-model tuning or breaks with certain quantization schemes, the practical barrier is higher than the paper suggests.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
KV² compression cuts long-context cache costs without sacrificing accuracy
KV² addresses a critical bottleneck in long-context LLM deployment: the memory cost of key-value caches when a single prefilled prompt must answer multiple queries. The technique uses a lightweight proxy to flag important tokens, then selectively recomputes only those for final scoring, avoiding the false choice between cheap-but-inaccurate compression and expensive full reprocessing. Results on…
MentionsQwen · CoMem · EncBank · Qwen3-8B
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Cache the Encoder Within:Compact, Reusable Memory across LLM Queries”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.