A Diagonal-Augmented quasi-Newton method with application to factorization machines

We present a novel quasi-Newton method for convex optimization, in which the Hessian estimates are based not only on the gradients, but also on the diagonal part of the true Hessian matrix (which can often be obtained with reasonable complexity). The new algorithm is based on the well known Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm and has similar complexity. The proposed Diagonal-Augmented BFGS (DA-BFGS) method is shown to be stable and achieves a super-linear convergence rate in a local neighborhood of the optimal argument. Numerical experiments on logistic regression and factorization machines problems showcase that DA-BFGS consistently outperforms the baseline BFGS and Newton algorithms.