Performances of low rank detectors based on random matrix theory with application to STAP

The paper addresses the problem of target detection embedded in a disturbance composed of a low rank Gaussian clutter and a white Gaussian noise. In this context, it is interesting to use an adaptive version of the Low Rank Normalized Matched Filter detector, denoted LR-ANMF, which is a function of the estimation of the projector onto the clutter subspace. In this paper, we show that the LR-ANMF detector based on the sample covariance matrix is consistent when the number of secondary data K tends to infinity for a fixed data dimension m but not consistent when m and K both tend to infinity at the same rate, i.e. m/K → c ∈ (0, 1). Using the results of random matrix theory, we then propose a new version of the LR-ANMF which is consistent in both cases and compare it to a previous version, the LR-GSCM detector. The application of the detectors from random matrix theory on STAP (Space Time Adaptive Processing) data shows the interest of our approach.