CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation
CrossPool addresses a critical infrastructure bottleneck in multi-model LLM serving: cold sparse MoE deployments waste GPU memory by reserving peak KV-cache capacity per model even when traffic remains sparse and uneven. The paper proposes disaggregating weights from transient KV-cache into separate memory pools, allowing dynamic reallocation of cache resources across underutilized models. This technique directly impacts production serving economics for providers hosting dozens of models with bursty, low-concurrency demand, improving both memory utilization and long-context support without architectural changes to existing inference stacks.
Modelwire context
Analyst takeThe paper's framing as a memory optimization paper undersells its actual competitive implication: providers running large catalogs of MoE models (think hosted fine-tune marketplaces or multi-tenant inference APIs) face a structural cost disadvantage today that CrossPool directly addresses, and the fix requires no changes to the inference stack itself.
This sits largely disconnected from the recent Modelwire coverage, which has concentrated on benchmarking, reasoning theory, and application-layer research. The closest adjacent thread is the RaDaR rare disease paper, which argued that compact specialized models can compete with frontier-scale ones on narrow tasks. CrossPool implicitly reinforces that argument from the infrastructure side: if serving many small specialized models becomes cheaper through dynamic KV-cache pooling, the economic case for deploying a fleet of domain-specific models (rather than one large generalist) gets meaningfully stronger. The two papers together sketch a plausible path toward specialized model proliferation that the field has discussed but not yet costed out.
Watch whether a major inference hosting provider (Fireworks, Together, or a hyperscaler inference API) cites or adopts CrossPool-style disaggregation within the next two quarters. Adoption at that layer would confirm the technique survives contact with real traffic distributions, not just the paper's synthetic workloads.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsCrossPool · MoE · KV-cache · GPU memory
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.