Asymptotic analysis of stochastic gradient-based adaptive filtering algorithms with general cost functions

This paper presents an analysis of stochastic gradient-based adaptive algorithms with general cost functions. The analysis holds under mild assumptions on the inputs and the cost function. The method of analysis is based on an asymptotic analysis of fixed stepsize adaptive algorithms and gives almost sure results regarding the behavior of the parameter estimates, whereas previous stochastic analyses typically considered mean and mean square behavior. The parameter estimates are shown to enter a small neighborhood about the optimum value and remain there for a finite length of time. Furthermore, almost sure exponential bounds are given for the rate of convergence of the parameter estimates. The asymptotic distribution of the parameter estimates is shown to be Gaussian with mean equal to the optimum value and covariance matrix that depends on the input statistics. Specific adaptive algorithms that fall under the framework of this paper are signed error least mean square (LMS), dual sign LMS, quantized state LMS, least mean fourth, dead zone algorithms, momentum algorithms, and leaky LMS.