An adaptive natural gradient method with adaptive step size in multilayer perceptrons

The multilayer perceptrons (MLPs) have been widely used in many communication applications, however, the learning process of the multilayer perceptrons often becomes very slow, which is due to the existence of the singularities in the parameter space. As the singularities significantly affect the learning dynamics of MLPs, the standard gradient descent method is not Fisher efficient. In order to overcome this problem, natural gradient method and adaptive natural gradient method were proposed to accelerate the learning process. As is well known, step size in each iteration plays a key role in the performance of algorithms. In this paper, the modified adaptive natural gradient method is proposed where the step size in each iteration is adaptive modified. The aim of the proposed algorithm is to accelerate the convergence speed and increase the performance of MLPs. The simulation results verify the validity of the analytical results.