Separating exploration from reward optimization in language model training

The key novelty is architectural: instead of adding exploration bonuses directly into the training objective (which prior work showed creates misaligned incentives), ExpDis runs exploration in a separate policy, then filters and distills only the correct trajectories back. This sidesteps the core problem identified in recent distillation research: that implicit reward signals can diverge from actual capability gains.

This directly addresses a problem surfaced in the October 2nd distillation collapse paper. That work showed on-policy distillation produces inconsistent results because teachers implicitly reward student behaviors regardless of quality, leading to gaming rather than genuine capability expansion. ExpDis solves this by enforcing a hard correctness filter before distillation occurs, ensuring the reward signal tracks real improvement. The approach also builds on the October 1st intrinsic rewards paper, which demonstrated that standard exploration bonuses often fail to gather decision-relevant experience. By isolating exploration into a separate stage and using verifiable rewards as the gate, ExpDis avoids the trap of exploration objectives that look productive but don't actually improve downstream performance.

If ExpDis produces reasoning patterns that supervised fine-tuning cannot discover on the same model family and dataset, and those patterns transfer to held-out tasks, that confirms the method finds genuinely new solution families rather than just recovering known strategies under different training dynamics. Watch for ablations showing performance drops if the correctness filter is removed or if exploration bonuses are reintroduced into the base model training loop.