Steepest Descent Algorithms in Optimization with Good Convergence Properties

There are interesting new algorithms which overcome the slow convergence near a minimum point of the standard steepest descent algorithm. For a convex quadratic function we have n-steps convergence if the step lengths are set equal to the reciprocals of the eigenvalues of the Hessian matrix. The Barzilai-Borwein step lengths are sometimes equal to the reciprocals of eigenvalues. It provides good convergence properties for quadratic functions. However with the BB-step lengths the function may increase as the iteration progresses. Also the BB-step lengths can be negative when it is applied to a nonlinear function which is nonconvex. We make some modifications in the use of the BB-step lengths to ensure monotonic decreases in the function as the iteration progresses. This leads to algorithms which have good convergence properties for a nonlinear function which can be nonconvex. We can prove global convergence for a function which has properly nested level sets by using the Inverse Lyapunov Function Theorem.