On the convergence of least-squares estimates

The authors consider the problem of convergence of least-squares (LS) estimates in a stochastic linear regression model. It is well known that if the parameter estimates are known to converge, the convergence analysis for many adaptive systems can be rendered considerably less arduous. For an important case where the regression vector is a measurable function of the observations and the noise is Gaussian, it has been shown, by using a Bayesian embedding argument, that the LS estimates converge almost surely for almost all true parameters in the parameter space. However, nothing can be said about a particular given system, which is usually the objective. It has long been conjectured that such a bad zero measure set in the parameter space does not actually exist. A conclusive answer is provided to this important question and it is shown that the set can indeed exist. This then shows that to provide conclusive convergence results for stochastic adaptive systems, it is necessary to resort to a sample pathwise analysis instead of the Bayesian embedding approach.<>