Strengthened Information-theoretic Bounds on the Generalization Error

The following problem is considered: given a joint distribution P
_XY
and an event E, bound P
_XY
(E) in terms of P
_X
P
_Y
(E) (where P
_X
P
_Y
is the product of the marginals of P
_XY
) and a measure of dependence of X and Y. Such bounds have direct applications in the analysis of the generalization error of learning algorithms, where E represents a large error event and the measure of dependence controls the degree of overfitting. Herein, bounds are demonstrated using several information-theoretic metrics, in particular: mutual information, lautum information, maximal leakage, and J
_∞
. The mutual information bound can outperform comparable bounds in the literature by an arbitrarily large factor.