Performance Improvements in Natural Gradient Descent Method

Optimization algorithms play a crucial role in many fields including artificial intelligence and condensed matter physics. However, achieving precise and fast convergence is still difficult for finding the global minimal value of a given cost function. The natural gradient method has demonstrated its effectiveness in accelerating convergence, particularly in deep learning, tensor networks, and variational quantum algorithms. Here, we enhance the natural gradient method by looking for a more suitable metric by introducing a ‘proper’ reference Riemannian manifold with Quasi-Newton methods. This integration reduces the computation and accelerates convergence in two numerical tests. Furthermore, our approach provides a novel way of combining algorithms that may shed more light on optimization methods.