SPSA Method Using Diagonalized Hessian Estimate

Simultaneous perturbation stochastic approximation (SPSA) and its adaptive version (ASPSA) are two commonly used methods in stochastic optimization problems, analagous to the gradient descent and Newton-Raphson methods in deterministic optimization. However, both methods have potential shortcomings. SPSA, as a first-order-type method, has typically rapid improvement in the early stages, but slow convergence at the later stages of the search process. ASPSA, as a second-order method, has typically faster convergence in the later stages, but a more numerically challenging implementation. We propose a method (diagSPSA or diagSG) using only diagonal elements of Hessian estimates to re-scale gradients when updating parameters in each iteration. This method uses part of the information of Hessian matrices and has low computational cost. We prove the convergence performance and asymptotic behaviors of diagSPSA. In addition, this paper presents a theoretical efficiency analysis, comparing the new method diagSG against stochastic gradient method (SG). We also make numerical tests for the efficiency of both diagSPSA and diagSG.