Regime switching stochastic approximation algorithms with switched ODE limit

Our objective is to treat tracking problems where the time-varying signal is a Markov chain with finite state space. The motivation stems from using recursive stochastic algorithms for tracking Markovian parameters such as those in spreading code optimization in CDMA wireless communication. We use constant step size algorithm to update the increments of a sequence of occupation measures. The usual stochastic approximation (SA) techniques cannot be carried over in the analysis due to the time-varying nature and Markovian jumps. Combining stochastic approximation method and two-time scale Markov chains, we develop asymptotic properties of the algorithm. The results are distinct from the usual SA results. We show that under suitable conditions, continuous-time interpolation of the iterates converges weakly to regime-switching ordinary differential equation. Then suitably scaled sequence of the tracking error is shown to converge to a switching diffusion. As an application of these results, the performance of an adaptive discrete stochastic approximation algorithm is analyzed.