By Topic

On improving trust-region variable projection algorithms for separable nonlinear least squares learning

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

2 Author(s)
Mizutani, E. ; Dept. of Ind. Manage., Nat. Taiwan Univ. of Sci. & Technol., Taipei, Taiwan ; Demmel, J.

In numerical linear algebra, the variable projection (VP) algorithm has been a standard approach to separable “mixed” linear and nonlinear least squares problems since early 1970s. Such a separable case often arises in diverse contexts of machine learning (e.g., with generalized linear discriminant functions); yet VP is not fully investigated in the literature. We thus describe in detail its implementation issues, highlighting an economical trust-region implementation of VP in the framework of a so-called block-arrow least squares (BA) algorithm for a general multiple-response nonlinear model. We then present numerical results using an exponential-mixture benchmark, seven-bit parity, and color reproduction problems; in some situations, VP enjoys quick convergence and attains high classification rates, while in some others VP works poorly. This observation motivates us to investigate original VP's strengths and weaknesses compared with other (full-functional) approaches. To overcome the limitation of VP, we suggests how VP can be modified to be a Hessian matrix-based approach that exploits negative curvature when it arises. For this purpose, our economical BA algorithm is very useful in implementing such a modified VP especially when a given model is expressed in a multi-layer (neural) network for efficient Hessian evaluation by the so-called second-order stagewise backpropagation.

Published in:

Neural Networks (IJCNN), The 2011 International Joint Conference on

Date of Conference:

July 31 2011-Aug. 5 2011