Direction of Arrival Estimation Based on a Mixed Signal Transmission Model Employing a Linear Tripole Array

Direction of arrival (DOA) estimation is an important topic in array signal processing. Currently, most research activities are focused on the single signal transmission (SST) type of signals, i.e. only one physical signal is used to carry the information from a transmitter to a receiver with a given polarisation setting. However, to make full use of the degrees of freedom in spatial domain, signals based on the dual signal transmission (DST) model are more and more widely used, i.e., two signals with different polarisations carrying different information are employed for communication between the transmitter and the receiver. But there is rarely any work on DOA estimation of DST signals. Motivated by such a problem, the paper proposes two methods for DOA estimation of signals based on a mixed signal transmission (MST) model, i.e., a mixture of SST and DST signals. The first method provides a two-step solution and estimate the DOA of the SST signals first and then the DST signals second. The second method estimates the DOA of all signals in one step. Moreover, CRB (Cramér-Rao Bound) for the estimation model is derived to evaluate the performance the proposed methods.


I. INTRODUCTION
Direction of arrival (DOA) estimation has been widely studied in recent years [1]- [6], and many algorithms have been introduced to solve the DOA estimation problem, such as multiple signal classification (MUSIC) [7]- [10], estimation of signal parameters via rotational invariance techniques (ESPRIT) [11]- [14] and those based on sparsity or compressive sensing (CS) [15]- [19]. In its early time, most research on DOA estimation was based on omnidirectional antennas, ignoring the polarisation information of impinging signals. To consider the polarisation information, electromagnetic (EM) vector sensor arrays were proposed to jointly estimate the DOA and polarisation information [20]- [25]. The MUSIC, ESPRIT and CS-based algorithms can be extended to solve the joint DOA and polarisation estimation problem [26]- [36]. However, in their models, for each direction, it is assumed either explicitly or implicitly that The associate editor coordinating the review of this manuscript and approving it for publication was Yanhui Liu . there is only one signal impinging upon the array; in other words, each source only emits one single signal with specific direction and polarisation and we refer to such a system as a single signal transmission (SST) system.
The SST signals have a fixed polarisation state, which will not change with time. Sometimes they are also referred to as fully-polarised (FP) signals [37]- [40]. However, due to reflection or some other channel effects, signals may have their polarisation states varying with time, which can be referred to as partially-polarised (PP) signals [37], [38]. In [39], [40], it is pointed out that the DOA estimation algorithms for FP signals are not applicable to PP signals and new algorithms are proposed to solve the problem in mixed signal scenario where source signals include both FP and PP signals. As introduced in [41], a PP signal can be viewed as a sum of unpolarised and fully polarised components. A similar case in wireless communications is the dual signal transmission (DST) model [42]- [48], where two separate SST/FP signals are transmitted simultaneously from each source, which makes full use of the degrees of freedom (DOFs) provided by a vector sensor array. For a DST signal, the two sub-signals have the same DOA but different polarisations and carry different information. One DST signal example is to use two orthogonal linearly polarized signals with amplitude or phase modulation [43], [44], [46], [47]. However, there has rarely been any research reported on estimating the DOAs of DST signals. Instinctively, we could consider a DST signal as two independent SST signals and estimate their DOAs one by one. However, as we will see later, a direct application of the traditional DOA estimation methods such as the subspace-based ones may not work as expected for DST signals and a new approach is needed.
In this work, based on a uniform linear tripole sensor array, we first try to extend the classic MUSIC algorithm straightforwardly to the four-dimensional (4-D) case to find the parameters of a mixture of impinging SST and DST signals, i.e., a mixed signal transmission (MST) model. As analysed later, due to inherent physical property of signal polarisation and array structure, we can only find the DOA and polarisation parameters of SST signals and for the DST signals, it fails completely for both DOA and polarisation parameters due to an ambiguity problem with their estimation. The ambiguity problem associated with the polarisation parameters of DST signals cannot be solved by any estimator due to limitation of the DOFs available in the polarisation domain. However, it is possible to obtain only the DOA information of DST signals (not polarisation information). As a solution and also to reduce the complexity of the 4-D searching process of the extended MUSIC algorithm and exploit the additional information provided by DST signals, i.e. the two sub-signals of a DST signal share the same DOA, a two-step algorithm is proposed first, which was published in our earlier conference paper and report [25], [49]. In this solution, the DOA and polarisation information of SST signals are found first by a rank-reduction algorithm (referred to as the SST estimator) and then the DOA information of the DST signals is estimated by a specifically designed estimator (referred to as the DST estimator). Then, a general estimator (referred to as the MST estimator) is proposed which can obtain the DOA parameters of the SST and DST signals in one single step, while the polarisation information of SST signals can be obtained by a separate two-dimensional (2-D) search if needed. A new complete detailed proof for the proposed method is provided which is not available in [25]. Moreover, the CRB (Cramér-Rao Bound) is derived to evaluate the performance of the proposed estimation algorithms. As demonstrated by simulation results, for SST signals, the two proposed estimators (the two-step estimator and the general MST estimator) have a similar performance, while the general estimator has a higher accuracy in estimating the direction of DST signals.
This paper is structured as follows. The MST signal model is introduced in Section II. In Section III, the traditional subspace based estimator is extended to the 4-D case, followed by the two-step method associated with SST and DST estimators and the unified one-step MST estimator. The CRB is derived in Section IV. Simulation results are provided in Section V, with conclusions drawn in Section VI.

II. SIGNAL MODELS
In our mixed signal transmission model, there are M 1 SST and M 2 DST narrowband non-linearly polarized sources impinging on a uniform linear array (ULA) with N tripole sensors from the far field as shown in Fig. 1. Each SST source emits only one signal s m (t), m = 1, 2, · · · , M 1 , and each DST source emits two sub-signals s M 1 +2m−1 (t) and s M 1 +2m (t), m = 1, 2, · · · , M 2 , with the same elevation-azimuth angle (θ, φ) but different polarisation (γ , η), where γ , η denote the polarisation auxiliary angle and the polarisation phase difference, respectively. For convenience, the parameters of the DST signal s M 1 +2m−1 and s M 1 +2m are denoted by (θ Moreover, a basic assumption is that the SST and DST signals come from different elevation-azimuth cone, where sin θ 1 sin φ 1 = sin θ 2 sin φ 2 . . . In discrete form, the received SST signals of a single tripole sensor at the k-th time instant is denoted by a 3×1 vector x s [k] (noise-free) where p m is the SST angular-polarisation vector given by In the above equation, m denotes the angular matrix associated with DOA parameters θ and φ, and g m is the polarisation vector including polarisation parameters γ and η, given by The DST signals collected by a single tripole sensor can be considered as the sum of all 2M 2 sub-signals, where each sub-signal can be viewed as an SST signal. Hence, the received DST signals are in the form Considering a pair of sub-signals as a single composite DST signal, we can use a 2 × 1 vector s m to denote the m-th DST signal corresponding to the pair of sub-signals s M 1 +2m−1 and s M 1 +2m , defined by Then, (7) can be transformed to where P m is the angular-polarisation matrix for DST signals, Note that the two sub-signals of the same DST signal share the same angular matrix, and here we use m to represent the common angular matrix of the m-th DST signal, i.e.
We use G m to denote the polarisation matrix of the m-th DST signal, defined as Then, P m is the product of m and G m , The total received signal x[k] is the sum of SST and DST signals, which is given by Now we consider the whole array system. The steering vector a m is given by (15) where τ = 2πd λ with d being the adjacent sensor spacing. Firstly, consider each-sub signal as a separate SST signal. With the Gaussian white noise n[k] of variance σ 2 n , the array output snapshot at the k-th time instant y[k] is given by [36] (16) where '⊗' is the Kronecker product. Consider each pair of sub-signals as a DST signal, (16) will be transformed to (17) In the DST signal part, as each pair of sub-signals comes from the same direction, the two steering vectors are equal to each other, i.e.
Then, a M 1 +2m−1 in (17) can also be replaced by a M 1 +2m . To further simplify (17), q m is used to denote the direction-polarisation joint steering vector for SST signals, q m = a m ⊗ p m (19) and Q m is the DST joint steering matrix, given by With the above notation, (17) is further changed to The covariance matrix of the received signals is given by For the general MST model, assume M 1 + 2M 2 < 3N . After eigendecomposition, R can be expressed by where λ n is the n-th eigenvalue and u n is the associated eigenvector. After sorting the 3N eigenvalues in descending order, the eigenvectors u 1 , u 2 , . . . , u M 1 +2M 2 form the signal subspace U s , while u M 1 +2M 2 +1 , u M 1 +2M 2 +2 , . . . , u 3N form the noise subspace U n .

III. PROPOSED ESTIMATORS
As mentioned in Introduction, it seems that the traditional subspace-based DOA estimation algorithms could be used to find the DOA of both SST and DST signals. Therefore, to show its limitation in DOA estimation for a mixture of SST and DST signals, we will first try to extend the classic MUSIC algorithm to the 4-D case. To overcome its limitation and also exploit the additional information carried by DST signals, a two-step algorithm is then proposed, consisting of two estimators, one for SST signals and one for DST signals.
After that the one-step general MST estimator is proposed.

A. EXTENSION OF THE TRADITIONAL MUSIC ESTIMATOR TO 4-D
A traditional DOA estimator considers all incoming signals as separate SST signals, i.e. the given M 1 SST signals and M 2 DST signals will be considered as M 1 + 2M 2 SST signals in the algorithm. Since the joint steering vectors q m are orthogonal to the noise subspace, then The DOA and polarisation parameters are estimated by finding the peaks of the following cost function through a 4-D search.
However, as shown in the following, there is an ambiguity problem with both DOA and polarisation of DST signals, which can not be obtained by the method in (25).
Firstly, the ambiguity problem associated with the polarisation parameters is analysed. Suppose that the two sub-signals of the DST signal come from (θ, φ, γ 1 , η 1 ) and (θ, φ, γ 2 , η 2 ), and they are also considered as two separate SST signals. a is used to denote the common steering vector of the two subsignals. p 1 and p 2 are used to denote their angular-polarisation vectors based on distinct polarisation parameters. According to (4), it can be obtained that Consider a non-existing signal from the same DOA (θ, φ) of the sub-signals above with an arbitrary polarisation (γ 3 , η 3 ) different from (γ 1 , η 1 ) and (γ 2 , η 2 ). The angular-polarisation vector p 3 can be denoted as From (4), it can be learned that g m , m ∈ [1, 3], is a column vector with two elements and is a matrix with two columns. Then, (26) and (27) can be changed to where ω 1 and ω 2 denote the first and second column vectors of . g m1 and g m2 are the first and second elements in g m , respectively. As ω 1 is not in parallel with ω 2 from (4), (28) indicates that p 1 , p 2 and p 3 are three vectors in the same 2-D space determined by ω 1 and ω 2 . That means there exists a linear relationship among p 1 , p 2 and p 3 , where λ 1 and λ 2 are constants. When the estimator is applied to a DST signal, the noise subspace will be orthogonal to the joint steering vectors of both sub-signals, where Then, we have As a result, F(θ , φ , γ 3 , η 3 ) will be recognised as a peak in the spectrum and wrongly identified as the parameters of a non-existing source. This means the algorithm fails when trying to estimate the polarisation of DST signals. Note that this is an inherent limitation for DST signals and there is no way to identify their polarisation parameters.
Next, we give an analysis to the ambiguity problem associated with the DOA of DST signals. (29) shows that by 4-D MUSIC, the DST signal's direction with arbitrary polarisation will be recognised as a false peak. The 'arbitrary polarisation' includes a special polarisation: the linear polarisation. As introduced in [50], there is an infinite number of ambiguity steering vectors in parallel with a linearly polarised signal, where the ambiguity directions are in linear polarisation as well. From (4), the angular-polarisation vector p m can be viewed as an arbitrary vector in a 2-D space constructed by the two column vectors of m , where the elements in g m denote the weights of the vectors to indicate how these two column vectors form p m . If a signal s 1 is linearly polarised, it means that the angular-polarisation vector p 1 is real-valued. As the two column vectors in 1 are both real-valued, the intersection vector between 1 and another different 2-D space 2 is also real-valued. Consequently, it is possible to locate the angular-polarisation vector p 1 as the intersection between 1 and 2 . This means that in the 2-D space 2 , there exists another angular-polarisation vector p 2 in parallel with p 1 . If a 1 is also in parallel with a 2 , for example, sin θ 1 sin φ 1 = sin θ 2 sin φ 2 while θ 1 = θ 2 , φ 1 = φ 2 , q 2 will be in parallel with q 1 and the parameters in q 2 will be recognised as a false peak.
For the scenario with mixed signals, although the 4-D search algorithm cannot identify the DST signals, it works for SST signals. However, an obvious problem is the significantly high computational complexity of the 4-D peak search process. In the next subsection, a two-step algorithm is proposed, which estimates the DOAs of SST and DST signals separately with a much lower complexity.

B. THE PROPOSED TWO-STEP METHOD
As indicated by the name, there are two steps for the proposed method. The first step is to apply a newly proposed SST VOLUME 9, 2021 estimator to obtain the DOA and polarisation of SST signals, while the second step is to apply a specifically designed DST estimator to find the DOA of DST signals.
By exploiting the orthogonality between the joint steering vector q m and the noise subspace U n , we have where C m = a m ⊗ m . For SST signals, there is only one polarisation vector g m from a specific direction (θ m , φ m ) satisfying [U H n C m ]g m = 0 and (32) indicates that the column rank of U H n C m equals By multiplying its Hermitian transpose on the right side, the product matrix is a 2 × 2 matrix with rank 1, i.e., As the matrix is not of full rank, we have where det{} represents the determinant of the matrix. Taking its inverse, a DOA estimator for SST signals is given by With the DOA information obtained, the polarisation parameters can then be estimated through another 2-D search using (32). Besides, the first step will also detect the desired DOA angles of DST signals but with an infinite number of ambiguity directions. In the next step, the DOA of DST signals will be extracted from these results. Since a DST signal s m consists of two sub-signals s M 1 +2m−1 and s M 1 +2m with different polarisations, we have Hence, the following cost function can be used to estimate directions of DST signals where || · || 2 denotes the l 2 -norm of the vector. When the above DST estimator is applied to a mixture of SST and DST signals, it only selects directions with However, for SST signals, (33) indicates that the rank of C H m U n U H n C m is 1 and therefore, as desired, the DST estimator in (37) will miss the SST signals.
A summary to the proposed two-step algorithm: • Calculate the noise subspace U n by applying eigenvalue decomposition to the estimated covariance matrixR.
• Apply the SST estimator (35) and find the DOAs of SST signals by 2-D search.
• Find the polarisation parameters of SST signals using (32) by 2-D search if needed.
• Apply (37) to estimate the DOAs of DST signals.
Note that a special case, if the source signals are only SST signals, the DOAs can be all obtained by (35) and there is no need to continue with the second step (37). In this scenario, the two-step method is degraded to the same as an existing low-complexity 2-D MUSIC algorithm.

C. THE PROPOSED GENERAL MST ESTIMATOR
Instead of employing separate estimators for SST and DST signals, a single general estimator for MST signals is proposed in this section.
Before introducing the general estimator, we first investigate the rank and determinant of the two matrices (a m ⊗ For convenience we drop the subscript m and denote the two matrices as where A is always a matrix with non-zero diagonal elements (proof in Appendix A). In the scenario with a mixture of SST and DST signals, we can divide the direction range into four regions: the SST signal direction region, the DST signal direction region, the DST ambiguity direction region and the remaining uninterested direction region. Table 1 gives a summary of the ranks of A and B and the their associated direction regions (see more details in Appendix B). As discussed before, the SST estimator selects the direction with the condition rank{B} < 2 and the DST estimator selects the direction with rank{B} = 0. From Table 1, we can see that the SST estimator can find the SST signal directions, DST signal directions and the DST ambiguity directions, while the DST estimator only estimates the DST signal directions. As a solution, we propose a general MST estimator which can work in all cases of signals and its cost function is given by where A 3,3 is the 2 × 2 cofactor matrix of A by removing its third row and third column. The estimator is able to estimate DOA information for all signals without determining its type, i.e. SST or DST. After obtaining all the DOAs, if needed, we can then use (32) to find the polarisation parameters of SST signals through 2-D search. For DST signals, we can distinguish them from the SST ones by checking whether there is polarisation ambiguity problem or not when performing the search using (32). The reason why the above cost function works can be explained as follows. First, matrix A is a 3 × 3 matrix and its cofactor matrix is of 2 × 2. For the SST signal direction region, since every two column vectors in U H n (a ⊗ I 3 ) are linearly independent (Appendix B, case 1) and the cofactor A 3,3 can be viewed as the product of the first and the second column vectors of U H n (a ⊗ I 3 ), the rank of A 3,3 should be 2 and the numerator det{A 3,3 } in (40) is nonzero. However, matrix B has a rank of 1 and its determinant is zero; as a result, the cost function at the directions of SST signals will have a peak (infinitely large in theory).
For the DST direction region, the rank of matrix A is 1 and then its cofactor matrix must have a rank of 1 and nonzero-valued (Appendix A). Although its determinant is zero, it approaches zero at those directions at the first order, while the 2 × 2 matrix B has a rank of 0 and its determinant is zero and approaches zero at those directions at the second order (a 2 × 2 zero matrix); as a result, the cost function at the directions of DST signals will have a peak too (an infinitely large value in theory).
For the DST ambiguity region, the rank of matrix A is 1 and similar to the case of DST direction region, the numerator of (40) is zero, but it approaches zero at those ambiguity directions at the first order, while the 2 × 2 matrix B has a rank of 1 and its determinant is zero and approaches zero at those directions at the first order (a 2×2 nonzero matrix); as a result, the cost function at the DST ambiguity region will be a nonzero finite value, but not a peak representing an infinitely large value.
For the uninterested region, both matrices A and B have full rank and neither of the numerator and denominator of the cost function is zero-valued; as a result, the cost function at this region will have a nonzero finite value, but not a peak representing an infinitely large value.
A detailed proof can be found in Appendix C.
A summary for the unified general MST estimator is given below: • Calculate the noise space U n by applying eigenvalue decomposition to the estimated covariance matrixR.
• Apply the MST estimator (40) to obtain the DOA of all signals by 2-D search.
• Find the polarisation parameters of SST signals by (32) if needed.

D. COMPLEXITY COMPARISON BETWEEN THE TWO PROPOSED METHODS
For the two-step method, the computation can be divided into two parts: the first is about working out the noise space (the eigenvectors of covariance matrix R). For a symmetric matrix, the complexity is O(n 3 ), where n is the dimension of the matrix. Here we take QR decomposition with Householder transform as an example. Since the dimension of R is 3N (N is the sensor number), 2/3 * (3N ) 3 = 18N 3 multiplications are needed in one iteration. Hence, 18kN 3 multiplications are needed in total with k iterations. (Note that the multiplications here are all complex-valued.) The second part is spectrum searching. In the first step of the method, estimator (35) requires 24N * (3N −M 1 −2M 2 )+2 multiplications in one search. If there are L searches in one direction, the total number of searches will be L 2 . Hence, the searching complexity of the first step is Overall, the two-step method's complexity is 18kN 3 Since the conventional MUSIC algorithm is the special case of the two-step method when source signals are all of SST, in this situation, we only need to apply the first step to estimate all DOAs of SST signals. The complexity is then 18kN 3 For the one-step method, the computation can also be divided into the eigenvector part and search part. The computation of eigenvectors is the same as the two-step method. The complexity of this part is also 18kN 3  In conclusion, the complexity of two proposed methods mainly depend on the sensor number N and the search number L 2 . If N is large enough and L 2 is not a very large number, the complexity of the two methods is both of O(N 3 ). They may have the same performance in operating time. However, in more practical scenarios, usually the sensor number N is rather limited and L 2 is very large to achieve a higher estimation accuracy; in this situation, the one-step method saves about half of the computation compared to the two-step method.

IV. CRAMÉR-RAO BOUND FOR MST SIGNALS
Now we derive the CRB for DOA estimation of a mixture of one SST signal and one DST signal to evaluate the performance of the proposed algorithms. A basic assumption is that all source signals are unconditional [51], [52], which means the source signals are random in all realizations. The SST signal and the two DST sub-signals have the same power σ 2 s . Here we use the symbol α to denote the parameters to be estimated, where (θ 1 , φ 1 ) is the DOA parameters for the SST signal and (θ 2 , φ 2 ) is the parameters for the DST signal. Note that in the DOA estimation process, the polarisation parameters can be considered as irrelevant parameters. From (16), the received VOLUME 9, 2021 signals can be changed to The equation holds because Define a matrix of the received signals Z[k], where For each snapshot, the probability density function is given by [1] where V Z (α) is the variance of Z and m(α) its mean value. The joint probability density function with K snapshots can be denoted as which leads to the following log-likelihood function (47) The elements in the Fisher information matrix (FIM) can be found as where i, j are integers and i, j ∈ [1,8].
According to (8.32) in [1], (48) can be simplified to Since the source signals are unconditional, we have m(α) = 0 (50) and The FIM elements are transformed to The FIM is a 4 × 4 matrix. The CRB for DOA information can be obtained as The CRB for SST signals only and DST signals only can be obtained by simply removing the DST or SST signal part in (44), and the FIM will be reduced to a 2 × 2 matrix.

V. SIMULATION RESULTS
In this section, simulations are performed based on a scenario with one SST signal and one DST signal impinging on the array from the far field. The SST estimator result is shown in Fig. 2, where the peak corresponding to the SST signal appears around the aimed direction (20 • , 20 • ); however, the DST signal direction is shown among a band of peak points instead of a single peak. On the other hand, the second step focuses on locating DST signals and as shown in Fig. 3, only a single peak appears around the aimed DST signal direction (30 • , 80 • ) while the SST signal direction is lost in the spectrum.   For the one-step general estimator or the so-called MST estimator, the spectrum has two peaks at around (θ, φ) = (20 • , 20 • ) and (30 • , 80 • ), indicating both directions have been identified successfully.

B. RMSE RESULT
In this part, the estimation accuracy of the two proposed solutions is compared in three scenarios: one SST signal only, one DST signal only, and a mixture of one SST signal and one DST signal. The directions of SST and DST signals are the same as in Section V-A, and the power of SST signal is equal to that of one DST sub-signal. We calculate the RMSE (root mean square error) of the azimuth-elevation angle (θ, φ) by 200 Monte-Carlo trials. The number of snapshots K = 100 and the searching step size is 0.05 • .
In the first scenario, only one SST signal impinges on the array. As shown in Figs. 5 and 6, the two methods have almost the same estimating accuracy and the RMSE of both estimators decreases gradually with the increasing SNR.
In the second scenario with DST signals, the results are presented in Figs. 7 and 8. Compared to the SST case, the DST case has lower average estimation errors, and the general one-step method has a higher accuracy than the two-step method.        the estimation error increases a little due to the additional DST signal. Figs. 11 and 12 also indicates the same difference versus Figs. 7 and 8. In this scenario, the two proposed methods still have a very similar performance in estimating the SST signal direction. However, the general one-step method has a lower RMSE than the two-step method with the DST signals.

VI. CONCLUSION
In this paper, the DOA estimation problem for a mixture of SST and DST signals has been studied based on a tripole linear array. Two subspace based DOA estimation methods were proposed and the CRB was derived to evaluate their performance. The two-step method estimates the SST and DST signals' directions separately with two corresponding estimators, one for the SST signals and one for the DST ones. The second method is a general one-step method which estimates the signal directions together without distinguishing the different types of signals. Simulation results showed that the two proposed methods have a very similar performance for SST signals, but the one-step method has some advantages in dealing with DST signals.

APPENDIX A PROOF OF NON-ZERO DIAGONAL ELEMENTS IN MATRIX A
For both SST and DST signals, based on (4), (19), (24) and (32), it can be obtained that which means that the noise subspace is orthogonal to the 3M ×1 column vector q m . As there are more than one impinging signals, the noise subspace should be only orthogonal to those column vectors that are related to each SST signal or DST sub-signal or linear combination of these column vectors.
Consider a direction (θ t , φ t , γ t , η t ) that is in the same elevation-azimuth angle cone with one impinging signal (θ 1 , φ 1 , γ 1 , η 1 ) while the four parameters are different. There must be no impinging signals from the assuming direction according to (2), where The column vector q t can be denoted as a linear combination of the three column vectors of matrix a t ⊗I 3 . Similarly, q 1 can also be considered as a linear combination of the vectors in a 1 ⊗I 3 . As a is only dependent on sin θ sin φ, a t = a 1 . With variation of (θ t , φ t , γ t , η t ), it can always be found that at least one (in fact an infinite number) direction (θ t , φ t , γ t , η t ) that satisfies q t ∦ q 1 . Then, we draw a conclusion that there exists at least one direction (θ t , φ t , γ t , η t ) that and every elements in U H n q t is non-zero valued. Further, where every column in U H n (a 1 ⊗I 3 ) must be a non-zero vector. From (39), firstly we expand matrix A by Dividing A p into column vectors, The diagonal elements of matrix A is the squared l 2 norm of each column vector in A p , which can be denoted as As discussed in (57), for impinging signals A p is a non-zero matrix and the inside column vectors are non-zero as well.
It can be concluded that the diagonal elements A m,m is real-valued and positive.

APPENDIX B RANK ANALYSIS OF TABLE 1
Here we divide the spatial spectrum into four regions: 1. the SST direction region; 2. the DST direction region; 3. the DST ambiguity direction region; 4. the uninterested region. We will analyse the listed regions with four cases. Case 1: Assuming an SST signal comes from (θ 1 , φ 1 , γ 1 , η 1 ), we have A new direction (θ 1 , φ 1 , γ 2 , η 2 ) can be obtained by changing the polarisation parameters. Obviously, if γ 1 = γ 2 or η 1 = η 2 , q 1 will not be in parallel with q 2 , where This means U H n C 1 = 0, but the two inner column vectors are linearly dependent. Thus, for the SST direction region, rank(B) = 1. Another new direction (θ 1 , φ 3 , γ 3 , η 3 ) can be obtained by only keeping the elevation angle unchanged, as introduced in Appendix A, and there exist more than one direction which makes q 1 not in parallel with q 3 . Then is not a zero-matrix. Besides, (62) indicates the column vectors in U H n (a 1 ⊗ I 3 ) are linearly dependent. However, as p is always a vector with non-zero elements (non-linearly polarised and θ = 0), the rank of the matrix U H n (a 1 ⊗ I 3 ) equals 2 and every two column vectors are linearly independent. Thus, it can be obtained that for the SST direction region, rank(A) = 2. Cases 2 and 3: In this case, the matrix rank of DST direction region and DST ambiguity region will be discussed together.
In a DST direction region, assume the two sub-signals are from (θ 1 , φ 1 , γ 1 , η 1 ) and (θ 1 , φ 1 , γ 2 , η 2 ). By (38), we have rank(B) = 0 and As mentioned, in the spectrum, the DST ambiguity directions, which are in the same elevation-azimuth zone with DST signals, may also produce peaks. The ambiguity direction can be denoted as (θ 3 , φ 3 , γ 3 , η 3 ). However, there is only one pair of linearly polarised (γ 3 , η 3 ) with the direction (θ 3 , φ 3 ). Thus, the direction (θ 3 , φ 3 ) with non-linear polarisation or other linear polarisation parameters will not be recognised as peaks, which means in this situation, we have Considering (65) and (66) together, it can be obtained that U H n (a 1 ⊗ I 3 ) is not a zero matrix and all the row vectors in this matrix must be orthogonal to both column vectors in 1 . As the two column vectors are two 3 × 1 vectors which are not in parallel with each other, the only explanation is that all the row vectors in U H n (a 1 ⊗ I 3 ) are in parallel with each other. As a result, the row rank of U H n (a 1 ⊗ I 3 ) equals 1. Then we know that for the DST direction region, rank(A) = 1.
Since the DST ambiguity direction has the same elevation angle as the DST direction, they have exactly the same matrix A. Thus, for DST ambiguity direction, rank(A) = 1. As discussed above, with the direction (θ 3 , φ 3 ), there is only one pair of linearly polarised (γ 3 , η 3 ) which may produce the false peak, where With the same direction (θ 3 , φ 3 ), the ambiguity will not occur with another pair of polarisation (γ 3 , η 3 ), which means Considering the two equations, it can be concluded that the row rank of U H n (a 1 ⊗ 3 ) equals 1. Thus, for the DST ambiguity region, rank(B) = 1.
Case 4: For the remaining region (θ, φ, γ , η) in the spectrum, the noise subspace is not orthogonal to the related joint steering vector. In this situation, matrices A and B both have a full rank, where rank(A) = 3 and rank(B) = 2.

APPENDIX C PROOF OF THE ONE-STEP ESTIMATOR
For convenience, we use x to denote sin θ and y to denote sin φ. Then, the steering vector a becomes where C = e −jτ is a constant. The angular matrix becomes Adding an infinitely small value x → 0 and y → 0 to x and y, respectively, the new steering vectorâ becomeŝ a = [1, C 2(xy+x y+y x+ x y) , . . . , C (N −1)(xy+x y+y x+ x y) ] T We useā to denote the difference between the two vectors a =â − a (72) VOLUME 9, 2021 Its n-th elementā n , n ∈ [1, N ] is expressed as a n = C (n−1)xy [C (n−1)(x y+y x+ x y) Similarly, the difference between the original and the new angular matrix¯ can also be calculated bȳ The differencesā and¯ lead to changes of matrices A and B. The changed matrices arê Replacing A, B byÂ,B in (40), we have where v and w are the determinant differences between the original and the changed matrices. When x, y → 0, v and w also approach 0. Now we consider the four cases listed in Table 1. Case 1: For SST signal directions, rank{A} = 2 and rank{B} = 1. As discussed in Section III-C, rank{A 3,3 } = 2, which means the determinant det{A 3,3 } must be non-zero.
As v → 0, the numerator of the estimator approaches a non-zero constant. Since rank{B} = 1, we have det{B} = 0. As w → 0, the denominator of the estimator approaches 0. Hence for SST signal directions, the estimator will have an infinitely large value and the directions will be detected by the estimator as peaks.
Case 2: Expanding (76), we havê For DST signals, a and T are vector and matrix with constant value elements. Then, we have the difference matrixĀ as Define D = C y x+x y+ x y . As x, y → 0, we have D → 1. By (73), the elementsĀ ij , i, j ∈ [1,3] can be expressed in the following form (ignoring the constant factor determined by i, j) where the symbol '↔' denotes the equation of the same infinitesimal order. The symbol 'O' denotes the infinitesimal order and E is defined as E = D − 1. Ignoring the high order infinitesimal, we haveĀ Comparing the infinitesimal order between E and y x+x y, then The equation shows E and y x + x y (x,y are constants) have the same infinitesimal order, and thus, Assume the original matrix A is in the form Here we take the first cofactor of A as an example, i.e., By adding the cofactor of difference matrixĀ 3,3 , we havê The determinant is given by where the components A 11 A 22 − A 12 A 21 = 0 because rank{A 3,3 } = 1 in the DST region.
The infinitesimal order ofB ij is determined by¯ ij . Takinḡ 11 in (75) as an example, the order of this infinitesimal can be calculated bȳ 11 The denominator is non-zero and can be ignored, and then, the equation becomes By ignoring the high order components above, we havē Similarly, the order of¯ 12 ,¯ 21 ,¯ 22 and¯ 31 is given bȳ By (91), we haveB where k 1 , k 2 are non-zero constants. Then, comparing the order betweenB ij and det{Â 3,3 }, we have lim x, y→0B ij det{Â 3,3 } = lim x, y→0 Obviously, the infinitesimal order of both denominator and numerator is determined by the infinitesimal with lower order between x and y. Besides, the limit in (97) does not exist because the nearby limits approach different constant values, which meansB ij has the same infinitesimal order with det{Â 3,3 }, where As DST signals have rank{B} = 0, B = 0. The determinant ofB is denoted by The infinitesimal order is The estimator is calculated as lim x, y→0 The infinity value indicates peaks in the DOA spectrum and the estimator can also find the DST signal directions successfully. Case 3: For DST ambiguity directions, rank{A} = 1 and rank{B} = 1. The numerator of the estimator is the same as in the DST signal direction case, which is denoted by However, the rank of matrix B is one, which means B is not a zero matrix and cannot be ignored.
In this case, the estimator is in the form of lim x, y→0 The final results will approach an undetermined constants instead of infinity. As a result, in the DOA spectrum, the DST ambiguity directions will not appear as a peak. Case 4: For the uninterested directions, rank{A} = 3 and rank{B} = 2, which means these two matrices are of full-rank. The numerator of the estimator det{Â 3,3 } will be a non-zero constant, and so is the denominator det{B}. The results of the estimator are finite values and these directions will not appear as peaks in the DOA spectrum.