Post-training cache routers cut MoE memory overhead without changing inference rules
Researchers propose a post-training framework that solves a critical bottleneck in MoE inference: GPU memory constraints that force repeated expert weight transfers during decoding. The approach jointly optimizes routing and cache management through two modes. Temporal Router predicts which experts will be reused in subsequent tokens and keeps them resident, while Spatio-Temporal Router refines predictions using causal context from prior layers. By treating cache optimization as a learnable problem rather than a fixed system constraint, this work enables larger MoE models to run on memory-limited hardware without sacrificing the native Top-K selection mechanism. Results on Qwen3 and GPT-OSS suggest practical gains for production inference workloads.
Modelwire context
ExplainerThe paper treats expert weight residency as a learnable optimization problem rather than accepting it as a fixed hardware constraint. Prior MoE work has focused on routing quality or expert decomposition, but this work makes the cache itself part of the training objective.
This sits directly alongside the PCoMoE and Contrastive Routing papers from early September, which also refined how MoE systems allocate computation. Where those papers optimized routing signals and expert granularity, this one optimizes what stays in memory during inference. The BeaconKV work on cache compression (same day) addresses a related bottleneck in reasoning models, but that work focuses on which tokens matter for attention, while this paper focuses on which expert weights to keep resident. Together, these three papers suggest the MoE inference stack is becoming a tuning surface rather than a fixed architecture.
If Qwen3 or GPT-OSS deployments report memory savings on long-context tasks (>8K tokens) within the next two quarters, that confirms the routing predictions generalize beyond the training distribution. If memory gains plateau or require retraining for new token lengths, the approach is narrower than claimed.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsQwen3 · GPT-OSS · Mixture-of-Experts · GSM8K
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.