Equivalence of Empirical Risk Minimization to Regularization on the Family of $f- \text{Divergences}$

The solution to empirical risk minimization with
$f-\mathbf{divergence}$
regularization
$(\mathbf{ERM}-f\mathbf{DR}$
) is presented under mild conditions on
$f$
. Under such conditions, the optimal measure is shown to be unique. Examples of the solution for particular choices of the function
$f$
are presented. Previously known solutions to common regularization choices are obtained by lever-aging the flexibility of the family of
$f-\mathbf{divergences}$
, These include the unique solutions to empirical risk minimization with relative entropy regularization (Type-I and Type-II). The analysis of the solution unveils the following properties of
$f-\mathbf{divergences}$
when used in the ERM-f DR problem:
$i$
)
$f-\mathbf{divergence}$
regularization forces the support of the solution to coincide with the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; and ii) any
$f-\mathbf{divergence}$
regularization is equivalent to a different
$f-\mathbf{divergence}$
regularization with an appropriate transformation of the empirical risk function.