Adaptive Target Detection and DOA Estimation With Uniform Rectangular Arrays in the Presence of Unknown Mutual Coupling

This paper investigates joint adaptive target detection and direction of arrival (DOA) estimation via a uniform rectangular array (URA) affected by mutual coupling. Capitalizing on a bespoke linearization of the array manifold and leveraging the banded symmetric Toeplitz block Toeplitz structure for the coupling matrix description, a vectorial model of the useful target echo return is proposed and used to formulate the detection problem. Two decision rules are designed, i.e., the generalized likelihood ratio (GLR) and multifamily likelihood ratio test (MFLRT), with the latter aimed at handling an unknown number of active mutual coupling coefficients. Both demand the joint maximum likelihood (ML) estimates of the coupling coefficients and the target angular displacement parameters which can be obtained solving a non-convex optimization problem. Toward this goal, an iterative procedure based on the minorization-maximization (MM) algorithm is developed. At the analysis stage, the performance of the proposed methods is assessed in terms of detection probability (
$P_{d}$
) and DOA root mean square error (RMSE) in comparison with benchmarks and standard strategies that do not account for the mutual coupling phenomenon. The results demonstrate the effectiveness of the proposed approaches to overcome signal mismatches induced by both the DOA uncertainty and mutual coupling.