A selective learning algorithm for nonlinear synapses in multilayer neural networks

In multilayer neural networks, network size reduction and fast convergence are important. For this purpose, trainable activation functions and nonlinear synapses have been proposed. When high-order polynomials are used for nonlinearity, the number of terms in the polynomial becomes very large for a high-dimensional input. It causes very complicated networks and slow convergence. In this paper, a method to select the useful terms in the polynomial in a learning process is proposed. This method is based on the genetic algorithm (GA), and combines the internal information and magnitude of connection weights to select the gene in the next generation. A mechanism of pruning the terms is inherently included. Many examples demonstrate the usefulness of the proposed method compared with the ordinary GA method. Convergence is stable and the number of the selected terms is well reduced.