Optimal Design of the Adaptive Normalized Matched Filter Detector Using Regularized Tyler Estimators

This paper addresses improvements on the design of the adaptive normalized matched filter (ANMF) for radar detection. It is well acknowledged that the estimation of the noise-clutter covariance matrix is a fundamental step in adaptive radar detection. In this paper, we consider regularized estimation methods that force by construction the eigenvalues of the covariance estimates to be greater than a positive regularization parameter ρ. This makes them more suitable for high-dimensional problems with a limited number of secondary data samples than traditional sample covariance estimates. The motivation behind this paper is to understand the effect and properly set the value of ρ that improves estimate conditioning while maintaining a low-estimation bias. More specifically, we consider the design of the ANMF detector for two kinds of regularized estimators, namely the regularized sample covariance matrix, the regularized Tyler estimator (RTE). The rationale behind this choice is that the RTE is efficient in mitigating the degradation caused by the presence of impulsive noises while inducing little loss when the noise is Gaussian. Based on asymptotic results brought by recent tools from random matrix theory, we propose a design for the regularization parameter that maximizes the asymptotic detection probability under constant asymptotic false alarm rates. Provided simulations support the efficiency of the proposed method, illustrating its gain over conventional settings of the regularization parameter.