On improving trust-region variable projection algorithms for separable nonlinear least squares learning

In numerical linear algebra, the variable projection (VP) algorithm has been a standard approach to separable “mixed” linear and nonlinear least squares problems since early 1970s. Such a separable case often arises in diverse contexts of machine learning (e.g., with generalized linear discriminant functions); yet VP is not fully investigated in the literature. We thus describe in detail its implementation issues, highlighting an economical trust-region implementation of VP in the framework of a so-called block-arrow least squares (BA) algorithm for a general multiple-response nonlinear model. We then present numerical results using an exponential-mixture benchmark, seven-bit parity, and color reproduction problems; in some situations, VP enjoys quick convergence and attains high classification rates, while in some others VP works poorly. This observation motivates us to investigate original VP's strengths and weaknesses compared with other (full-functional) approaches. To overcome the limitation of VP, we suggests how VP can be modified to be a Hessian matrix-based approach that exploits negative curvature when it arises. For this purpose, our economical BA algorithm is very useful in implementing such a modified VP especially when a given model is expressed in a multi-layer (neural) network for efficient Hessian evaluation by the so-called second-order stagewise backpropagation.