Evaluation of the Influences of Hyper-Parameters and L2-Norm Regularization on ANN Model for MNIST Recognition

Artificial neural networks (ANNs) have achieved the state-of-the-art performance in many tasks. Although ANNs do not require manually engineered features, there are lots of hyper-parameters to be optimized. The choices of hyper-parameters significantly impact the performance of the model. A number of methods like manual, grid, random search, and Bayesian optimization using Gaussian Processes (GPs) have been proposed to do the optimization. However, the work of exploring the individual effects of each parameter is rarely conducted. In this paper, we highlight on the effects of each individual hyper-parameter (involving the learning rate, batch size, the regularization item, and the number of iterations) on the model accuracy. We firstly construct a performance evaluation architecture that is based on method of normalized scoring to quantify the model performance. Then we establish an ANN for MNIST recognition and classification task to validate the proposed performance evaluation architecture. The experimental results show that a small learning rate and a small batch size not only affect the model's convergence speed but also make the model difficult to converge. The L2-norm regularization is critical for the overfitting elimination. Such an evaluation architecture can also be promoted to other similar related research areas.