Rademacher penalties and structural risk minimization
Koltchinskii, V.
Dept. of Math. & Stat., New Mexico Univ., Albuquerque, NM;
This paper appears in: Information Theory, IEEE Transactions on
Publication Date: Jul 2001
Volume: 47,
Issue: 5
On page(s): 1902-1914
ISSN: 0018-9448
References Cited: 39
CODEN: IETTAW
INSPEC Accession Number: 6980674
Digital Object Identifier: 10.1109/18.930926
Posted online: 2002-08-07 00:27:35.0
Abstract
We suggest a penalty function to be used in various problems of
structural risk minimization. This penalty is data dependent and is
based on the sup-norm of the so-called Rademacher process indexed by the
underlying class of functions (sets). The standard complexity penalties,
used in learning problems and based on the VC-dimensions of the classes,
are conservative upper bounds (in a probabilistic sense, uniformly over
the set of all underlying distributions) for the penalty we suggest.
Thus, for a particular distribution of training examples, one can expect
better performance of learning algorithms with the data-driven
Rademacher penalties. We obtain oracle inequalities for the theoretical
risk of estimators, obtained by structural minimization of the empirical
risk with Rademacher penalties. The inequalities imply some form of
optimality of the empirical risk minimizers. We also suggest an
iterative approach to structural risk minimization with Rademacher
penalties, in which the hierarchy of classes is not given in advance,
but is determined in the data-driven iterative process of risk
minimization. We prove probabilistic oracle inequalities for the
theoretical risk of the estimators based on this approach as well
Index
Terms
Available to subscribers and IEEE members.
References
Available to subscribers and IEEE members.
Citing Documents
Available to subscribers and IEEE members.
Cart
