Linear Two-Time-Scale Stochastic Approximation A Finite-Time Analysis

We consider two-time-scale stochastic approximation for finding the solution of a linear system of two equations. Such methods have found broad applications in many areas, especially in machine learning and reinforcement learning. A critical question in this area is to analyze the convergence rates (or sample complexity) of this method, which has not been fully addressed in the existing literature. Our contribution in this paper is, therefore, to provide a new analysis for the finite-time performance of the two-time-scale stochastic approximation. Our key idea is to leverage the common techniques from optimization, in particular, we utilize a residual function to capture the coupling between the two iterates. This will allow us to explicit design the two step sizes used by the two iterations as well as to provide a finite-time error bound on the convergence of the two iterates. Our analysis in this paper provides another aspect to the existing techniques in the literature of two-time-scale stochastic approximation, which we believe is more elegant and can be more applicable to many scenarios.