Using information theoretic learning techniques to train neural networks

This paper describes a method to train a multilayer perceptron (MLP) neural network using Information Theoretic Learning (ITL) techniques. The backpropagation algorithm, which is used to train MLPs, seeks to minimize the mean square error (MSE) between the output of the neural network and the target values. This paper describes a method to train MLPs by utilizing the minimum error entropy (MEE) of the error samples. The MSE is a second order statistic whereas the MEE uses the probability density function of the error samples. Therefore, the MEE technique uses higher order statistical information from the error samples to adapt the weights of the neural network. When the error distribution is non-gaussian, higher order statistical information can lead to faster training and smaller residual training error.