Training Time Prediction for Mixed Precision-based Distributed Training

The 147% prediction error figure is the buried lede here. Existing cost-estimation tools were built when FP32 was the default, so they treat precision as a constant rather than a variable, which means any cluster scheduling or budget forecast built on those tools is systematically wrong the moment you introduce BF16 or FP8 into the mix.

This connects most directly to the optimizer benchmarking work covered yesterday ('Benchmarking Optimizers for MLPs in Tabular Deep Learning'), which also exposed a gap between what practitioners assume about training efficiency and what empirical measurement actually shows. Both papers are pointing at the same underlying problem: the tooling researchers use to reason about training costs lags behind the actual diversity of modern training configurations. The sparse attention work in AdaSplash-2 (covered April 16) adds another dimension here, since attention kernel precision choices are exactly the kind of variable a static computation graph model would mishandle.

Watch whether major cloud training platforms (AWS, Google Cloud, CoreWeave) integrate precision-aware cost estimation into their job schedulers within the next two quarters. If they do, this line of research has cleared the gap from academic result to operational tooling.