Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding

The core insight here is that the bottleneck in speculative decoding isn't the drafting method itself but the inability to predict, in advance, which method will work best for a given input. HVD treats that prediction as a learnable problem rather than a static configuration choice, which is a different framing than most inference optimization work.

This sits in a cluster of research attacking inference efficiency from different angles. Where the Majestic Labs Prometheus server story (covered June 1) represents a hardware-first approach to the memory wall throttling token generation, HVD is a software-layer response to the same underlying constraint: getting more output per unit of compute. The two approaches aren't in competition so much as they address different parts of the stack. The masked diffusion work from DSL-LLaDA and the D3IM sampler papers, also from May 31, are tackling parallel decoding inefficiencies in non-autoregressive models, which is a related but structurally distinct problem. HVD is squarely about autoregressive inference optimization.

The real test is whether the learned predictor generalizes across model families not represented in the three LLMs used here. If a follow-up evaluation on a significantly different architecture, say a mixture-of-experts model like Mellum2, shows comparable routing accuracy, the approach has legs beyond its training distribution.