Statistical characterization of the largest DMR-whitened eigenvalue for source enumeration

The goal of source enumeration is to estimate the number of sources impinging on an array of sensors. This paper considers the problem of source enumeration in colored noise, which is often encountered in sonar applications. When the background noise is colored, the standard approach is to whiten the data prior to estimating the number of sources. The whitening filter is designed using an estimate of the sample covariance matrix (SCM) derived from a training set containing only noise. In many underwater array applications, the number of snapshots available to estimate the noise-only SCM is limited due to environmental variability. Using results from random matrix theory, Nadakuditi and Silverstein proposed a source enumeration method that requires less data than previous approaches [1]. This paper explores a source enumeration method based on the Dominant Mode Rejection (DMR) whitening transform that can be implemented with even less data than Nadakuditi and Silverstein's approach. Specifically, this paper presents a statistical characterization of the largest DMR-whitened eigenvalue. Simulations show that the largest DMR-whitened eigenvalue can be characterized by the Tracy-Widom distribution. This paper derives the parameters used to map the DMR eigenvalues to the Tracy-Widom distribution and shows how to set the detection threshold to guarantee a desired false alarm rate for the source enumeration algorithm.