Adaptive beamformer orthogonal rejection test

Research in the area of signal detection in the presence of unknown interference has resulted in a number of adaptive detection algorithms. Examples of such algorithms include the adaptive matched filter (AMF), the generalized likelihood ratio test (GLRT), and the adaptive coherence estimator (ACE). Each of these algorithms results in a tradeoff between detection performance for matched signals and rejection performance for mismatch signals. This paper introduces a new detection algorithm we call the adaptive beamformer orthogonal rejection test (ABORT). Our test decides if an observation contains a multidimensional signal belonging to one subspace or if it contains a multidimensional signal belonging to an orthogonal subspace when unknown complex Gaussian noise is present. In our analysis, we use a statistical hypothesis testing framework to develop a generalized likelihood ratio decision rule. We evaluate the performance of this decision rule in both the matched and mismatched signal cases. Our results show that for constant power complex Gaussian noise, if the signal is matched to the steering vector, ABORT, GLRT, and AMF give approximately equivalent probability of detection, higher than that of ACE, which trades detection probability for an extra invariance to scale mismatch between training and test data. Of these four tests, ACE is most selective and, therefore, least tolerant of mismatch, whereas AMF is most tolerant of mismatch and, therefore, least selective, ABORT and GLRT offer compromises between these extremes, with ABORT more like ACE and with GLRT more like AMF.