On the normalization of the step size in nonsymmetric stochastic gradient algorithms

The normalization of the step size in the least-mean-square (LMS) algorithm allows easy control of the range of stable operation for the normalized step size in the normalized LMS (NLMS) algorithm, and also easy determination of the step size for maximum convergence speed. A generalized step size normalization for stochastic gradient algorithms in which the gradient vector differs from the data vector is proposed. Three applications are considered in detail: the stochastic Newton scheme, the sign-data LMS algorithm, and an instrumental variable method proposed to speed up the convergence of the LMS algorithm.<
>