A Review of Optimization Algorithms for Training Neural Networks

The selection of the optimization algorithm (optimizer) is one of the most essential endeavors in Deep Learning and across all categories of Neural Networks. It's a matter of experimenting by making mistakes and learning from them. In this research, we will conduct experiments using 7 of the most common optimization algorithms, namely: Stochastic Gradient Descent, Root Mean Square Propagation, Adaptive Gradient Algorithm, Adadelta, Adaptive Moment Estimation, Adamax, and Nadam on MNIST datasets, to determine which one provides the deep neural network with the highest accuracy and performance. A data scientist will benefit from the insightful analysis provided by this study in selecting the most appropriate optimizer to use while modeling their deep neural network.