On the PDF Estimation for Information Theoretic Learning for Neural Networks

Information-Theoretic Learning (ITL) is one of the new methods gaining popularity used for adaptive signal processing learning algorithms and has many advantages compared to traditional method which minimizes the mean square error (MSE). Previously [12], we described a method based on the backpropagation algorithm to train a type of neural network called the multi-layer perceptron (MLP) using Information Theoretic Learning (ITL) techniques. Our method was developed to train MLPs by utilizing the minimum error entropy (MEE) of the error samples. The MSE is a second order statistic whereas the MEE uses the probability density function of the error samples. Therefore, the MEE technique uses higher order statistical information from the error samples to adapt the weights of the neural network. When the error distribution is non-gaussian, higher order statistical information can lead to faster training and smaller residual training error. The Probability Density Function (PDF) estimation using the Parzen window could affect the accuracy of the Back-Propagation training. In this paper, we investigate the effects of the Parzen Window estimator on the efficacy of the ITL training using Renyi's Entropy and Shannon's Entropy. Using different estimators and simulations, we compare MLP using the typical backpropagation algorithm (using MSE and cross-entropy) and also one using ITL methods in terms of convergence speed of the weights, PDF estimator and the residual error. We use standard data sets (like the MNIST handwriting data set available on the Internet) to train and test the MLP using all these methods. Simulation results compare the prediction accuracy of the three different types of backpropagation algorithms (MSE, Shannon's cross-entropy, Renyi's quadratic entropy) in the paper.