A Representation-Focused Training Algorithm for Deep Networks

We present an optimization algorithm for the training of deep networks that focuses the optimization on the parameters of the feature representation while eliminating the optimization on the parameters of the prediction map. The proposed algorithm can be analyzed for least-squares objectives for which the implicit minimization can be obtained analytically on mini batches. We illustrate the numerical performance of the optimization algorithm with convolutional networks and explore a variant using Adam as a wrapper algorithm.