On the resolution probability of conditional and unconditional maximum likelihood DOA estimation

The outlier production mechanism of maximum likelihood direction-of-arrival estimators is investigated. The objective is to provide an accurate description of the probability of resolution for both conditional and unconditionalmaximum likelihood methods in the small sample size regime. To that effect, the asymptotic behavior of these two cost functions is analyzed assuming that both the number of antennas and the number of available snapshots increase without bound at the same rate, so that both quantities are comparable in magnitude. The finite dimensional distributions of both conditional and unconditional cost functions are shown to be Gaussian in this asymptotic regime, and a closed form expression of the corresponding asymptotic covariance matrices is provided.