Accelerated Scaled Gradient Descent Algorithm For Low-Rank Matrix Factorization

Low-rank matrix factorization is a particularly effective method which finds numerous applications in signal processing, machine learning and imaging science. Recent literature has proposed that scaled gradient descent (ScaledGD) algorithm. It achieves linear convergence without depending on the condition number, while maintaining the low per-iteration cost. The task of this paper is to propose an accelerated ScaledGD algorithmic approach called scaled momentum algorithm, which using a method similar to momentum to accelerate ScaledGD. Momentum is a commonly used optimization algorithm in deep learning that helps accelerate gradient descent in the relevant direction and dampens oscillations. The new scaled momentum algorithm has a benign convergence guarantee similar to the ScaledGD that is independent of the number of conditions, while prominently improve the rate of convergence than ScaledGD.