NOAH: Newton Method of Optimization with Approximated Hessian

Newton method of optimization is very much useful in machine learning and deep learning optimizations due its order two convergence. The major problems of Newton method of optimization are the high cost of computing and storing the Hessian and its inverse for large-scale settings, which motivate us to use approximated Hessian instead of the true Hessian. However, existing works on Hessian approximation-based Newton methods are either computationally expensive or require more memory compared to first-order methods. To overcome these issues, this paper proposes NOAH, which approximates the Hessian with the help of first row of the true Hessian and directly finds the update step by using Sherman-Morrison formula of matrix inversion, which leads to faster convergence with comparatively less time and space complexities. Like Adam, NOAH uses exponential moving averages of the gradient and the Hessian curvature information to control the fluctuation of the update direction in heterogeneous settings. Experimental results show that, NOAH outperforms stochastic gradient descent (SGD) with momentum and state-of-the-arts Newton method of optimizations, L-BFGS and Nystrom-SGD in term of faster convergence with less time and space complexities.