Generalizing Nesterov’s Acceleration Framework by Embedding Momentum Into Estimating Sequences: New Algorithm and Bounds

We present a new type of heavy-ball momentum term, which is used to construct a class of generalized estimating sequences. These allow for accelerating the minimization process by exploiting the information accumulated in the previous iterates. Combining a newly introduced momentum term with the estimating sequences framework, we devise, as an example, a new black-box accelerated first-order method for solving smooth unconstrained optimization problems. We prove that the proposed method exhibits an improvement over the rate of the celebrated fast gradient method by at least a factor of $\frac{1}{{\sqrt 2 }}$, and establish that lower bound on the number of iterations carried through until convergence is $\mathcal{O}\left( {\sqrt {\frac{\kappa }{2}} } \right)$. Finally, the practical performance benefits of the proposed method are demonstrated by numerical experiments.