Strengthening Gradient Descent by Sequential Motion Optimization for Deep Neural Networks

In this article, we explore the advantages of heuristic mechanisms and devise a new optimization framework named sequential motion optimization (SMO) to strengthen gradient-based methods. The key idea of SMO is inspired from a movement mechanism in a recent metaheuristic method called balancing composite motion optimization (BCMO). Specifically, SMO establishes a sequential motion chain of two gradient-guided individuals, including a leader and a follower to enhance the effectiveness of parameter updates in each iteration. A surrogate gradient model with low computation cost is theoretically established to estimate the gradient of the follower by that of the leader through chain rule during the training process. Experimental results in terms of training quality on both fully connected multilayer perceptrons (MLPs) and convolutional neural networks (CNNs) with respect to three popular benchmark datasets, including MNIST, Fashion-MNIST, and CIFAR-10 demonstrate the superior performance of the proposed framework in comparison with the vanilla stochastic gradient descent (SGD) implemented via backpropagation (BP) algorithm. Although this study only introduces the vanilla gradient descent (GD) as a main gradient-guided factor in SMO for deep neural network (DNN) training application, it is great potential to combine with other gradient-based variants to improve its effectiveness and solve other large-scale optimization problems in practice.