2-D DOA Estimation for Incoherently Distributed Sources Considering Mixed Circular and Noncircular Signals in Massive MIMO System

In practical environment, the received signals cannot be regarded as point sources due to the existence of signal scattering phenomena. Incoherently distributed source model is suitable for this realistic scenarios. The incoherently distributed DOA estimators based on noncircular property can provide a better performance. However, these estimators are not fit for the mixed circular and noncircular signals. In this paper, a conjugate generalized shift invariance algorithm is proposed for incoherently distributed sources comprising of circular and noncircular signals. The general array manifold (GAM) model of incoherently distributed source is established by one-order Taylor series approximation. To utilize the noncircular property, each incoherently distributed circular source is separated as two strictly incoherently distributed noncircular sources. A generalized shift invariance property is derived by dividing the whole uniform rectangular array (URA) into four overlapping subarrays. Then, a modified conjugate generalized least square (CGLS) ESPRIT algorithm is proposed to obtain the central DOAs, which can improve the estimation accuracy without any spectrum searching. In addition, a pair matching method and an angular spreads estimation algorithm are also designed based on the GAM model for this mixed sources scenario. Furthermore, the Cramer–Rao bound (CRB) of proposed algorithm is given and analyzed under the mixed incoherently distributed sources scenario. Numerical results illustrate the validity of the proposed algorithm in this realistic scenario.