Fractional Order Gradient Descent with Caputo Derivatives for Product-Unit Neural Networks

As neural network architectures grow deeper, the choice and refinement of optimization algorithms play a pivotal role in enhancing training efficiency and improving generalization performance. Traditional gradient descent methods often suffer from that the convergence is slow and it is easily trapped in local minima, particularly when dealing with high dimensional and complex data. In this study, we propose a gradient descent algorithm based on Caputo fractional-order derivatives to optimize product unit neural networks, and give a comparison of the optimization paths of Caputo fractional order and traditional gradient descent. By addressing the limitations of conventional integer order optimization methods in capturing long term dependencies and non-local features, the introduction of Caputo fractional order derivatives leverages the exponential feature combination properties of product-units, leading to more stable convergence and greater robustness to noise in function approximation tasks. The classification results, computational complexity, and training time of gradient descent and Caputo fractional-order gradient descent are compared on five benchmark datasets, and the experiments show that Caputo fractional order gradient descent improves accuracy at the cost of increased complexity.