Variable Order Fractional Gradient Descent Method and Its Application in Neural Networks Optimization

In this paper, a novel variable order fractional gradient descent optimization algorithm is proposed, which generalizes the classical gradient descent method by introducing a kind of variable order fractional derivative. The derivative order is adjusted with the number of iterations. The convergence of the algorithm is analyzed. And the proposed method is also applied to the training of the full connected back propagation neural networks. Finally, numerical experiments are carried out to illustrate the effectiveness and practicability of proposed method. Results show that the proposed method has a faster learning rate and higher accuracy both in the training and validation sets.