Smoothed Hinge Loss and ℓ1 Support Vector Machines

A standard approach to solving the Support Vector Machine (SVM) optimization problem is to solve the dual problem, typically using a coordinate descent algorithm. When solving the dual problem, however, the compute load increases with the number of data points. Some approaches, such as Pegasos [1] have had great success solving the primal problem, rather than the dual. We follow the idea of solving the primal, rather than dual, problem but introduce a new algorithm that makes minimal use of the gradient and uses approximate second-order information in the form of an approximate Hessian. In particular, we present a new algorithm for solving the soft-margin SVM optimization problem with an ℓ
¹
penalty, called SmSVM. This algorithm is designed to require a modest number of passes over the data and can achieve an optimal solution with a minimal number of gradient calculations. The algorithm uses a smooth approximation for the hinge-loss function, and an active set approach for the ℓ
¹
penalty. We use the active set approach to make implementation optimizations by taking advantage of the feature selection to reduce the problem size of our matrix-vector and vector-vector linear algebra operations. These optimizations result in substantially faster training times, in addition to providing feature selection. Our experiments show impressive results, beating LIBLINEAR [2] in many scenarios both in test accuracy and training time. Our solutions also benefit from sparsity, thanks to the ℓ
¹
regularizer.