Introducing Information Metrics for Statistical Signal Processing [Lecture Notes]

Statistical signal processing involves extracting information from random observations, a task that relies heavily on metrics that quantify the information present in the data. These metrics are crucial for practitioners and researchers in fields such as radar, sonar, and communications. Over time, several key information metrics have been introduced, often using different names. This tutorial describes essential information metrics—the Kullback–Leibler divergence (KLD), Fisher information (FI), and mutual information (MI)—exploring their relationships and demonstrating their relevance to estimation and detection problems. The article illustrates these concepts using the example of a constant-amplitude signal in white Gaussian noise (WGN) and discusses the asymptotic behavior of these metrics in more general scenarios. Additionally, it presents the minimum discrimination information (MDI) theorem, which states that the probability density function (PDF) minimizing the KLD under a mean constraint belongs to the linear exponential family. The tutorial concludes with an exploration of the conditionality principle, revealing the insightful relationship among the KLD, signal-to-noise ratio (SNR), and MI and its implications for improved detection in a multipath environment.