Second-order Derivative Optimization Methods in Deep Learning Neural Networks

Optimization technique in deep learning neural networks is crucial to improve the parameters learning capability. Due to the recent advancements in technology and computational power, second-order derivative optimization methods have become increasingly popular as they offer additional information regarding the curvature of an objective function. In addition, multiple strategies can be applied based on the Hessian computation and step-length controlling parameter derived from the optimization trajectory when training a deep neural network. This paper demonstrates the latest second-order derivative optimization methods and their corresponding weight update rules in chronological order. The reviewed methods comprise Newton’s method, conjugate gradient, Quasi-Newton, Gauss-Newton, Hessian-free, Kronecker-factored approximate curvature, stochastic diagonal approximate greatest descent, AdaHessian, complex-step directional derivative, and Shampoo algorithms. A comparative study in terms of advantages, limitations, and performance are highlighted among these second-order derivative methods. An overall conclusion and future works are drawn in the paper to provide insights into the future development of optimization techniques.