DM-SARAH: A Variance Reduction Optimization Algorithm for Machine Learning Systems

Nowadays, the variance reduction (VR) technique is used to improve the performance of gradient-type algorithms in machine learning and deep learning. However, some existing VR algorithms require unrealistic assumptions or conditions such as τ-gradient dominated and Polyak-Lojasiewicz (PL) conditions, which limit their applications. In this paper, we present a Double Mini-batch StochAstic Recursive grAdient algoritHm (DM-SARAH) without these assumptions or conditions to solve the convex and non-convex optimization problems respectively. The main contributions of this paper are twofold: (1) At the theoretical level, we optimize the convergence rate and provide a complexity analysis of DM-SARAH, and (2) At the experimental level, we evaluate the effectiveness and efficiency of the proposed algorithm on various datasets. The experimental results indicate that the proposed algorithm outperforms existing methods.