MLE of Banded, Spiked, High-Dimensional Clutter Covariance–A Convex Relaxation Approach

In high-dimensional Space-Time Adaptive Processing (STAP), accurate estimation of the clutter-plus-noise covariance matrix is challenging, especially in low-sample regimes where traditional estimator like the Sample Covariance Matrix (SCM) becomes unstable. To address this challenge, we model the clutter-plus-noise covariance matrix as a “Banded+Spiked” covariance matrix. This model assumes that the clutter covariance matrix has a banded structure, while the singular values of the clutter-plus-noise covariance matrix are predominantly influenced by the noise floor. This aligns with a spiked covariance model, which effectively captures the dominant spiked characteristics of the clutter-plus-noise covariance matrix. We demonstrate that the Maximum Likelihood Estimation (MLE) problem for this estimation task is non-convex and therefore we propose a convex relaxation of the MLE. Our relaxation is formulated as a Frobenius loss minimization with non-smooth convex constraints that enforce banded sparsity in the covariance matrix. The proposed optimization algorithm jointly estimates both the banded clutter covariance matrix and the noise power. Numerical results using the high-fidelity RFView radar simulation environment show that our algorithm achieves a higher Signal-to-Clutter-plus-Noise Ratio (SCNR) compared to existing state-of-the-art methods, including TABASCO, Spiked Covariance Stein Shrinkage, and Diagonal Loading.