Stochastic Momentum Method With Double Acceleration for Regularized Empirical Risk Minimization

Momentum acceleration technique is famously known for building gradient-based algorithms with fast convergence in large-scale optimization. Recently, Nesterov 's momentum and Katyusha momentum have significantly improved the convergence for stochastic optimization problems. However, the practical gain of acceleration with Nesterov's momentum is mainly a by-product of mini-batching, while acceleration merely with Katyusha momentum in stochastic steps would make the optimization unstable. In this paper, we build a stochastic and doubly accelerated momentum method (SDAMM) which incorporates the Nesterov's momentum and Katyusha momentum in the framework of variance reduction, to stabilize the accelerated algorithm and reduce the dependence on the mini-batching. Theoretically, SDAMM achieves the best-known convergence rates for convex objectives. The experimental results demonstrate that our SDAMM is competitive with state-of-the-art methods for the optimization problems in machine learning.