On design of superlinear first order automatic machine learning techniques

Due to the computational excess of second order methods machine learning techniques in general, and neural network training techniques in particular, primarily employ first order line search optimization methods. The article presents a variation of first order line search optimization techniques that has superlinear convergence rates, i.e. the fastest convergence rates for first order methods. The presented algorithm has substantially simplified a line search subproblem into a single step calculation of the appropriate values of step length and/or momentum term. This remarkably simplifies the implementation and computational complexity of the line search subproblem and yet does not harm the stability of the methods. The algorithm is theoretically proven to be convergent, with superlinear convergence rates, and exactly classified within the newly proposed classification framework for first order techniques. Performance of the proposed algorithm is practically evaluated on five data sets and compared to the relevant standard first order optimization techniques. The results indicate superior performance of the presented algorithm over the standard first order methods.