On almost-sure bounds for the LMS algorithm

Almost-sure (a.s.) bounds for linear, constant-gain, adaptive filtering algorithms are investigated. For instance, under general pseudo-stationarity and dependence conditions on the driving data {/spl psi/k,k=1,2,3,...}, {Y/sub k/,k=0,1,2,...} a.s. convergence and rates of a.s. convergence (as the algorithm gain /spl epsiv//spl rarr/0) are established for the LMS algorithm h/sub k+1//sup /spl epsiv//=h/sub k//sup /spl epsiv//+/spl epsiv/Y/sub k/(/spl psi//sub k+1/-Y/sub k//sup T/h/sub k//sup /spl epsiv//) subject to some nonrandom initial condition h/sub 0//sup /spl epsiv//=h/sub 0/. In particular, defining {g/sub k//sup /spl epsiv//}/sub k=0//sup /spl infin// by g/sub 0//sup /spl epsiv//=h/sub 0/ and g/sub k+1//sup /spl epsiv//=g/sub k//sup /spl epsiv//+/spl epsiv/(E[Y/sub k//spl psi//sub k+1/]-E[Y/sub k/Y/sub k//sup T/]g/sub k//sup /spl epsiv//) for k=0,1,2,..., the author shows that for any /spl gamma/>0 max/sub 0/spl les/k/spl les//spl gamma//spl epsiv/(-1/)|h/sub k//sup /spl epsiv//-g/sub k//sup /spl epsiv//|/spl rarr/0 as /spl epsiv//spl rarr/0 a.s. and under a stronger dependency condition, the author shows that for any 00, max/sub 0/spl les/k/spl les//spl gamma//spl epsiv/(-/spl zeta//)|h/sub k//sup /spl epsiv//-g/sub k//sup /spl epsiv//| converges (as /spl epsiv//spl rarr/0) a.s. At a rate marginally slower than O((/spl epsiv//sup 2-/spl zeta//log log(/spl epsiv//sup -/spl zeta//))/sup 1/2 /). Then, under a stronger pseudostationarity assumption it is shown that similar results hold if the sequences {g/sub k//sup /spl epsiv//}/sub k=0//sup /spl infin//,/spl epsiv/>0 in the above results are replaced with the solution g/sup 0/(/spl middot/) of a nonrandom linear ordinary differential equation, i.e. one has, max/sub 0/spl les/k/spl les/[/spl gamma//spl epsiv/(-/spl zeta//])|h/sub k//sup /spl epsiv//-g/sup 0/(/spl epsiv/k)|/spl rarr/0 as /spl epsiv//spl rarr/0 a.s., where one can attach a rate to this convergence under the stronger dependency condition. The almost-sure bounds contained in the paper complement previously developed weak convergence results in Kushner and Shwartz [IEEE Trans. Information Theory, IT-30(2), 177-182, 1984] and, as are "near optimal". Moreover, the proofs used to establish these bounds are quite elementary.<
>