Regularizing Multilayer Perceptron for Generalization Using KL-Divergence

Generalization implies the ability to recognize an input pattern not used during the training of a neural network. Effective regularization strategies are one of the ways to make the generalization capability of the neural network better. In this article, the objective is to improve the generalization of multilayer perceptron (MLP) by regularization with the help of KL-Divergence (KLD) based weight adaption technique. For the purpose, we modify the conventional weight adaption criterion of MLP by incorporating KLD as an additional regularization term with the mean squared error function. We have tested the proposed regularizing approach on ten benchmark classification data sets. Our experimental results show that the KLD based regularized MLP provides better classification accuracy compared to the conventional MLP. We also observe that the efficiency of the proposed method is highly dependent on the coefficient (β) of the regularization term. The proposed method shows the best result for β = 0.02 for eight out of ten classification data sets.