Revisiting Auxiliary Losses for Conditional Depth Routing: An Empirical Study

The paper's real contribution isn't a new loss function but a warning: combining predictive and explicit supervision losses without careful tuning can destabilize training in ways that aren't obvious from looking at either loss in isolation. That interaction effect is the finding, not the individual loss comparisons.

Conditional depth routing is essentially a learned gating mechanism that decides how much compute to spend on each token, which puts this work in the same cluster as inference efficiency research. The SpecGuard paper covered here on April 16 ('From Tokens to Steps') also targets compute allocation at inference time, though through speculative decoding rather than architectural routing. The connection is thematic rather than direct: both are trying to spend fewer cycles on tokens that don't need them. This routing paper operates at training time and model architecture, while SpecGuard operates post-training, so they address different points in the pipeline and don't obviously compete.

The study uses a 157.5M-parameter decoder trained on fineweb-edu. If a follow-up replicates the instability findings at 1B-plus parameters with a different dataset mix, the interaction effects are a genuine architectural concern worth designing around. If they don't replicate at scale, this may be a regime-specific artifact.