Bias Compounds, Variance Washes Out

Stochastic rounding, which introduces random unbiased errors that cancel out over time, enables low-precision BF16 optimizer states to match full-precision FP32 performance, while deterministic round-to-nearest rounding causes repeated errors that compound and stall training. The key insight is that biased errors grow linearly with steps while unbiased errors grow as the square root, making bias elimination critical for long training runs. Implementing stochastic rounding in optimizer kernels adds no memory or computational overhead, making it a practical solution for memory-efficient training.