Improving generalisation with Ockham's networks: minimum description length networks

There exists a substantial problem in obtaining good generalisation performance in the application of artificial neural network technology where training data is limited. A number of current techniques aiming to improve generalisation are introduced from the perspective of the minimum description length (MDL) principle. These are quadratic weight decay, soft weight-sharing and the technique introduced by the authors, Ockham's networks. In addition to presenting the major developments of Ockham's networks, a summary of a case study comparing these techniques is presented. It is found that Ockham's networks provide an improvement in generalisation performance as good as any other technique tested in addition to using the smallest number of weights.<
>