Empirical study of PROXTONE and PROXTONE+ for fast learning of large scale sparse models

PROXTONE is a novel and fast method for the optimization of large scale non-smooth convex problems [1]. In this work, we try to use the PROXTONE method in solving large scale non-smooth non-convex problems, for example training of sparse deep neural networks (sparse DNN) or sparse convolutional neural networks (sparse CNN) for embedded or mobile device. PROXTONE converges much faster than first order methods, while first order methods are easy to derive and control the sparseness of the solutions. Thus in some applications, in order to train sparse models fast, we propose to combine the merits of both methods, that is we use PROXTONE in the first several epochs to reach the neighborhood of an optimal solution, and then use the first order method to explore the possibility of sparsity in the following training. We call such method PROXTONE plus (PROXTONE+). Both PROXTONE and PROXTONE+ are tested in our experiments, and which demonstrate both methods improved convergence speed twice as fast at least on diverse sparse model learning problems, and at the same time reduce the size to 0.5% for DNN models. The source of all the algorithms is available upon request.