T-LRA: Trend-Based Learning Rate Annealing for Deep Neural Networks

As deep learning has been widespread in a wide range of applications, its training speed and convergence have become crucial. Among different hyperparameters existed in the gradient descent algorithm, the learning rate has an essential role in the learning procedure. This paper presents a new statistical algorithm for adapting the learning rate during the training process. The proposed T-LRA (trend-based learning rate annealing) algorithm is calculated based on the statistical trends seen in the previous training iterations. The proposed algorithm is computationally very cheap and applicable to online training for very deep networks and large datasets. This efficient, simple, and well-principled algorithm not only improves the deep learning results, but also speeds up the training convergence. Experimental results on a multimedia dataset and deep learning networks demonstrate the effectiveness and efficiency of the proposed algorithm.