Variance Reduced Stochastic Optimization for PCA and PLS

Principal Component Analysis(PCA) is a dimensionality reduction technique which extracts the representative components of the data. Partial Least Squares(PLS) models the covariance structure between a pair of data matrices. The objective function of the two problems are similar and thus can often be solved by identical algorithms. Deterministic methods suffer from prohibitive computation cost in large-scale applications, while stochastic algorithms fail to achieve high-accuracy solutions. In this paper, we propose a new stochastic optimization method with variance reduction to solve PCA and PLS. Our method ensures to obtain high-accuracy solutions with enough computational cost, and rapidly converges to an approximate optima with few iterations. Extensive experiments demonstrate the significant performance of our method.