Rademacher penalties and structural risk minimization
Koltchinskii, V.
Information Theory, IEEE Transactions on
Volume 47, Issue 5, Jul 2001 Page(s):1902 - 1914
Digital Object Identifier 10.1109/18.930926
Summary:We suggest a penalty function to be used in various problems of
structural risk minimization. This penalty is data dependent and is
based on the sup-norm of the so-called Rademacher process indexed by the
underlying class of functions (sets). The standard complexity penalties,
used in learning problems and based on the VC-dimensions of the classes,
are conservative upper bounds (in a probabilistic sense, uniformly over
the set of all underlying distributions) for the penalty we suggest.
Thus, for a particular distribution of training examples, one can expect
better performance of learning algorithms with the data-driven
Rademacher penalties. We obtain oracle inequalities for the theoretical
risk of estimators, obtained by structural minimization of the empirical
risk with Rademacher penalties. The inequalities imply some form of
optimality of the empirical risk minimizers. We also suggest an
iterative approach to structural risk minimization with Rademacher
penalties, in which the hierarchy of classes is not given in advance,
but is determined in the data-driven iterative process of risk
minimization. We prove probabilistic oracle inequalities for the
theoretical risk of the estimators based on this approach as well
View citation and abstract |
Cart
