The Improved Adaptive Algorithm of Deep Learning with Barzilai-Borwein Step Size

To solve the problem that it is difficult to determine the learning rate when training a neural network model, this paper proposes an improved adaptive algorithm based on the Barzilai-Borwein (BB) step size. In this paper, the new algorithm accelerates the model's training through the second-order momentum and adapts the learning rate according to the BB step size. We also set an adequate range for the learning rate to ensure the stability of adaptive adjustment and reduce the error of step size. Compared with different algorithms in a series of popular models, the new algorithm significantly avoids the tediousness of manually adjusting the learning rate and helps to improve the convergence speed. The results show that the new algorithm is feasible and effective.