Optimal Design of Large Dimensional Adaptive Subspace Detectors

This paper addresses the design of adaptive subspace matched filter (ASMF) detectors in the presence of a mismatch in the steering vector. These detectors are coined as adaptive in reference to the step of utilizing an estimate of the clutter covariance matrix using training data of signal-free observations. To estimate the clutter covariance matrix, we employ regularized covariance estimators that, by construction, force the eigenvalues of the covariance estimates to be greater than a positive scalar ρ. While this feature is likely to increase the bias of the covariance estimate, it presents the advantage of improving its conditioning, thus making the regularization suitable for handling high-dimensional regimes. In this paper, we consider the setting of the regularization parameter and the threshold for ASMF detectors in both Gaussian and compound Gaussian clutters. In order to allow for a proper selection of these parameters, it is essential to analyze the false alarm and detection probabilities. For tractability, such a task is carried out under the asymptotic regime in which the number of observations and their dimensions grow simultaneously large, thereby allowing us to leverage existing results from random matrix theory. Simulation results are provided in order to illustrate the relevance of the proposed design strategy and to compare the performances of the proposed ASMF detectors versus adaptive normalized matched filter (ANMF) detectors under mismatch scenarios.