Probability of Resolution of Partially Relaxed Deterministic Maximum Likelihood: An Asymptotic Approach

The Partial Relaxation approach has recently been proposed for high resolution Direction-of-Arrival estimation (see Trinh-Hoang et al., IEEE Trans. Signal Process., 2018; Trinh-Hoang et al., ICASSP, 2018). In this article, we investigate the outlier production mechanism of the Partially Relaxed Deterministic Maximum Likelihood (PR-DML) Direction-of-Arrival estimator using tools from Random Matrix Theory. Instead of applying a single source approximation to multiple-source estimation criteria, which is the case for the MUSIC algorithm, the conventional beamformer, or the Capon beamformer, the Partial Relaxation approach accounts for the existence of multiple sources. In the Partial Relaxation framework, the structure of the desired direction is kept, whereas the sensor array manifold corresponding to the remaining signals is relaxed. This procedure allows to compute a closed-form solution for the relaxed signal part and to come up with a simple spectral search. In this article, an accurate description of the probability of resolution for the PR-DML estimator is provided by analyzing the asymptotic behavior of the PR-DML cost function, assuming that both the number of antennas and the number of snapshots increase without bound at the same rate. The finite dimensional distribution of the PR-DML cost function is shown to be Gaussian in this asymptotic regime and this result is used to compute the probability of resolution.