A merging and splitting algorithm based on mutual information for design neural networks

This paper presents a new algorithm based on the theory of mutual information and the relationship between the neural networks' architecture and learning capacity, called adaptive merging and splitting algorithm (AM SA), in designing optima feedforward neural networks (FNNs). This algorithm merges and splits hidden neurons during the training process of FNNs. Unlike most previous studies, AMSA puts emphasis on autonomous functioning in the design process of FNNs. This is the main reason why AMSA uses an adaptive not a predefined fixed strategy in designing FNNs. The adaptive strategy merges or splits hidden neurons based on the learning ability of hidden neurons or the training progress of FNNs. In order to reduce the amount of retraining after modifying FNN architectures, AMSA prunes hidden neurons by merging correlated hidden neurons and adds hidden neurons by splitting existing hidden neurons. In merge operation, the hidden neurons is merged based on mutual information criterion, resulting in compact neural networks architecture without decreasing the information processing capacity. In split operation, a new hidden neuron is added by splitting a hidden neuron in current neural networks, resulting in improve the information processing capacity. AMSA has been tested on the regression problem. The experimental results show that AMSA can design compact FNNs architectures with good generalization ability compared to other algorithm.