The Generalized Chernoff-Stein Lemma, Applications and Examples

A generalized notion of “relative entropy typicality” is introduced. The new definition accommodates non-i.i.d. scenarios and is parameterized by two families of parameters, $\left\{\delta^{[n]}\right\}_{n}$ and $\left\{\epsilon^{[n]}\right\}_{n}$: the choice of $\left\{\delta^{[n]}\right\}_{n}$ (the allowed “deviation” in the relative entropy typical set) can be optimized as a function of $\left\{\epsilon^{[n]}\right\}_{n}$, where ($1-\epsilon^{[n]}$) is the desired probability of the set. This generalized definition is shown to yield an extension of the Chernoff-Stein lemma, which characterizes the rate of decay of the type II error in hypothesis testing (under a fixed type I error constraint) as the KL divergence. In particular, the generalization accommodates both discrete and continuous random variables, non-i.i.d. settings, in addition to cases where the KL divergence grows non-linearly. Several example applications are discussed, including testing two correlated Gaussian distributions.