CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation
Source published ·Modelwire updated
Original coverage: arXiv cs.LG ↗·How Modelwire adds context
The development
CrossPool addresses a critical infrastructure bottleneck in multi-model LLM serving: cold sparse MoE deployments waste GPU memory by reserving peak KV-cache capacity per model even when traffic remains sparse and uneven. The paper proposes disaggregating weights from transient KV-cache into separate memory pools, allowing dynamic reallocation of cache resources across underutilized models. This technique directly impacts production serving economics for providers hosting dozens of models with bursty, low-concurrency demand, improving both memory utilization and long-context support without architectural changes to existing inference stacks.
Modelwire’s AI-generated summary of coverage from arXiv cs.LG.
Modelwire analysis
Analyst takeOur AI-generated reading of the wider context and the next developments to watch.
The paper's framing as a memory optimization paper undersells its actual competitive implication: providers running large catalogs of MoE models (think hosted fine-tune marketplaces or multi-tenant inference APIs) face a structural cost disadvantage today that CrossPool directly addresses, and the fix requires no changes to the inference stack itself.
This sits largely disconnected from the recent Modelwire coverage, which has concentrated on benchmarking, reasoning theory, and application-layer research. The closest adjacent thread is the RaDaR rare disease paper, which argued that compact specialized models can compete with frontier-scale ones on narrow tasks. CrossPool implicitly reinforces that argument from the infrastructure side: if serving many small specialized models becomes cheaper through dynamic KV-cache pooling, the economic case for deploying a fleet of domain-specific models (rather than one large generalist) gets meaningfully stronger. The two papers together sketch a plausible path toward specialized model proliferation that the field has discussed but not yet costed out.
Watch whether a major inference hosting provider (Fireworks, Together, or a hyperscaler inference API) cites or adopts CrossPool-style disaggregation within the next two quarters. Adoption at that layer would confirm the technique survives contact with real traffic distributions, not just the paper's synthetic workloads.
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
A specialized reasoning large language model for accelerating rare disease diagnosis: a randomized AI physician assistance trial
RaDaR demonstrates a strategic shift toward domain-specialized reasoning models that outperform much larger open-source alternatives on clinical tasks. The 32B parameter model, trained on 49K real cases plus 104K synthetic reasoning-enhanced examples, addresses a critical gap in rare disease diagnosis where expert scarcity creates diagnostic delays. This work signals that compact, task-specific LLMs with structured…
MentionsCrossPool · MoE · KV-cache · GPU memory
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.