Adaptive Low-Rank Gradient Descent

Low-rank structures have been observed in several recent empirical studies in many machine and deep learning problems, where the loss function demonstrates significant variation only in a lower dimensional subspace. While traditional gradient-based optimization algorithms are computationally costly for high-dimensional parameter spaces, such low-rank structures provide an opportunity to mitigate this cost. In this paper, we aim to leverage low-rank structures to alleviate the computational cost of first-order methods and study Adaptive Low-Rank Gradient Descent (AdaLRGD). The main idea of this method is to begin the optimization procedure in a very small subspace and gradually and adaptively augment it by including more directions. We show that for smooth and strongly convex objectives and any target accuracy
$\epsilon$
, AdaLRGD's complexity is
$\mathcal{O}(r\ln(r/\epsilon))$
for some rank
$r$
no more than dimension
$d$
. This significantly improves upon gradient descent's complexity of
$\mathcal{O}(d\ln(1/\epsilon))$
when
$r\ll d$
. We also propose a practical implementation of AdaLRGD and demonstrate its ability to leverage existing low-rank structures in data.