Performance of the GLRT for adaptive vector subspace detection

The problem of adaptively detecting a signal confined to a given vector subspace in interference modeled as a zero-mean complex Gaussian N-vector is considered. The correlation properties of interference are not known but are estimated from a given set of secondary (or reference) vectors. The dimension of the known signal subspace is N/sub s/, where 1⩽N/sub s/⩽N. The Generalized Likelihood Ratio Test (GLRT) is cast in a slightly different setting to show that it belongs to a class of invariant tests. The maximal invariants for the class of invariant tests are identified and the joint probability density function of the maximal invariants under both the null hypothesis H/sub 0/ and the alternate hypothesis H/sub 1/ are derived. These expressions are used to show that for 1⩽N/sub s/