Training Neural Networks with Momental Bound of Learning Rate

Adaptive algorithms are widely used in deep learning because of their fast convergence. Among them, Adam is the most widely used algorithm. However, studies have shown that Adam's generalization ability is weak. AdaX is a variant of Adam, which modifies the second moment of Adam and has good generalization ability. We propose a new adaptive and momental bound algorithm, called AdaXod, which characterizes of exponentially averaging the learning rate and is particularly useful for training deep neural networks. By setting an adaptively limited learning rate in the AdaX algorithm, the resultant AdaXod can effectively eliminate the problem of excessive learning rate in the later stage of neural network model training and thus stabilize training. Simulation experiments verify that AdaXod eliminates large learning rates during neural network training and outperforms other optimizers, especially on the complex network structures such as DenseNet.