An Exact Expression for the Gap in the Data Processing Inequality for $f$ -Divergences

When two input discrete distributions pass through a memoryless channel, the data processing inequality expresses the fact that the f divergence decreases. As is well known, in the case of the Kullback Leibler divergence, the chain rule makes it possible to obtain an exact expression for the gap in the data processing inequality, as a conditional divergence. In this paper, we provide a general expression for the gap valid for any convex function f , which generalizes the well-known formula for the Kullback-Leibler case. The result is a sum of Bregman divergences, which shows non-negativity. We then show how this identity may be used to give simple proofs of two classical results in linear algebra.