|
1. |
Model Selection via Bilevel Optimization
Bennett, K.P.; Jing Hu; Xiaoyun Ji; Kunapuli, G.; Jong-Shi Pang;
Neural Networks, 2006. IJCNN '06. International Joint Conference on
16-21 July 2006
Page(s):1922
-
1929
Abstract:
A key step in many statistical learning methods used in machine learning involves solving a convex optimization problem containing one or more hyper-parameters that must be selected by the users. While cross validation is a commonly employed and widely accepted method for selecting these parameters, its implementation by a grid-search procedure in the parameter space effectively limits the desirable number of hyper-parameters in a model, due to the combinatorial explosion of grid points in high dimensions. This paper proposes a novel bilevel optimization approach to cross validation that provides a systematic search of the hyper-parameters. The bilevel approach enables the use of the state-of-the-art optimization methods and their well-supported softwares. After introducing the bilevel programming approach, we discuss computational methods for solving a bilevel cross-validation program, and present numerical results to substantiate the viability of this novel approach as a promising computational tool for model selection in machine learning.
|
Cart
