Neural network training based on quasi-Newton method using Nesterov's accelerated gradient

This paper describes a novel quasi-Newton (QN) based accelerated technique for training of neural networks. Recently, Nesterov's accelerated gradient method has been utilized for training the neural networks. In this paper the acceleration of the QN training algorithm is realized using Momentum term in Nesterov's method. It is shown that the proposed algorithm has a similar convergence property with the conventional QN method. Neural network training for a simple function approximation and a microwave circuit modeling is presented to demonstrate the proposed algorithm. The proposed algorithm drastically improves the convergence speed of the QN algorithm.