Improved generalization in multi-layer perceptrons with the log-likelihood cost function

In supervised training of neural networks, synaptic weights are usually updated by an iterative algorithm which searches for the minimum of some cost function. The most common choice of cost function is the sum of squares (SS). An alternative choice of cost function is the log likelihood (LL). An analytical comparison of the SS and LL has suggested that the latter should lead to improved generalization when a multilayer perception trained on non-separable data by back-propagation. This is confirmed by results from simulated data, where the LL-trained networks result in a significant reduction in test-set errors. Moreover, a good generalizing solution appears to be achievable in fewer iterations, with a smaller network, or with fewer training samples.<
>