Routing gap partly explained by label noise, not router failure

The paper doesn't actually improve router performance; it argues the gap itself is partly illusory because single-draw evaluation conflates stochastic noise with systematic underperformance. The practical implication is that best-of-K sampling (a test-time workaround, not a router improvement) can mask weak routing, which flips the optimization priority from 'build better routers' to 'accept noise and sample more.'

This echoes the methodological skepticism in the RF drone benchmarks paper from early July, which exposed how evaluation splits can hide overfitting through data leakage. Here, the authors argue that standard routing evaluation similarly conflates measurement noise with model limitation. Both papers challenge whether published performance gaps reflect real capability shortfalls or evaluation artifacts. However, this routing work is narrower: it doesn't claim the gap is fake, only that practitioners may be misallocating effort by chasing router improvements when sampling strategies are cheaper. The cost-efficiency reframing matters for production systems, but the decomposition itself remains unvalidated on held-out routing benchmarks.

If the same decomposition framework is applied to a production routing system (e.g., Anthropic's or OpenAI's multi-model inference setup) and best-of-K sampling actually reduces per-token cost without degrading latency SLAs, the framing holds. If practitioners adopt this and find that label noise was overstated relative to systematic router bias, the paper's premise collapses. Watch for follow-up work that quantifies the noise-to-bias ratio on real-world routing datasets within the next six months.