Finite-Time Analysis of Markov Gradient Descent

Motivated by its broad applications in system identifications, stochastic control, and machine learning, we study the popular stochastic gradient descent (SGD) when the gradient samples of the underlying objective function are generated from Markov processes. This Markov sampling leads to the gradient samples being biased and not independent. The existing results for the convergence of SGD under Markov settings are often established under the assumption on the boundedness of either the iterates or the gradient samples. This assumption can be guaranteed through an impractical projection step to a compact set often defined over the unknown optimal solution of the underlying problem. In this article, we show that this projection step is unnecessary in many settings. In particular, we study the convergence of SGD under Markov samples without requiring any projection step for different objective functions, ranging from strongly convex to nonconvex objectives satisfying Polyak–Łojasiewicz conditions. We show that SGD converges nearly at the same rate with Markovian gradient samples as with independent gradient samples. The only difference is a logarithmic factor that accounts for the mixing time of the Markov process. Finally, we provide numerical experiments in robust identification problems to illustrate our theoretical results, where we consider SGD with samples generated by a Markov process.