First-Order Theory of Probabilistic Independence and Single-Letter Characterizations of Capacity Regions

This paper shows that many notions in information theory (e.g. entropy, capacity regions) can be defined using probabilistic independence alone. We consider the first-order theory of random variables with the probabilistic independence relation, which concerns statements consisting only of random variables, the probabilistic independence symbol, logical operators, and existential and universal quantifiers. Although probabilistic independence is the only non-logical relation included, this theory is surprisingly expressive, and is able to express notions such as entropy and cardinality, and interpret the true first-order arithmetic over natural numbers (and hence is undecidable). We also characterize the capacity region for a general class of multiuser coding settings (including broadcast channel, interference channel and relay channel) using a first-order formula, which can be regarded as a “single-letter characterization” of the capacity region of the aforementioned settings (conventional single-letter characterizations are existential formulae, whereas our formula contains both existential and universal quantifiers). We then introduce the linear entropy hierarchy to classify single-letter characterizations according to their complexity.