Tradeoffs Between Convergence Speed and Noise Amplification in First-Order Optimization: The Role of Averaging

We study the effect of averaging on convergence speed and noise amplification of the heavy-ball algorithm in the presence of additive white stochastic disturbances. For strongly convex quadratic problems, we show that averaging over the entire algorithmic history eliminates steady-state variance of the averaged output at the expense of slowing down convergence to a sub-linear rate. In contrast, finite window averaging converges with a linear rate but it leads to a non-zero value of the steady-state variance. While this value is smaller than the steady-state variance of the iterates of the heavy-ball algorithm, it has the same orderwise dependence on the condition number. We also show that the finite window averaging increases the upper bound on the expected error at iteration
$t$
by a constant factor that depends on the length of the averaging window.