Differentiation of neuron types by evolving activation function templates for artificial neural networks

In this paper we investigate the use of neuron-specific activation functions (AFs) within generalized multilayer perceptrons (GMLP). We utilize the netGEN system not only to evolve the structure of an artificial neural network (ANN), but also to search for a set of AF templates which are assigned to specific neurons by evolution. This may be seen as a loose analogy to neuron differentiation in biological neural networks (BNNs). While BNNs employ different neuron types in functionally different brain areas, neuron differentiation in ANNs might be useful to increase the adaptability to specific problems. The evolution of AF templates is based on evolving the control points of a cubic spline function, hence nonmonotonous AFs of (nearly) arbitrary shape may be generated. We present a number of experiments evolving ANN structure and AF templates using the parallel netGEN system to train the evolved architectures. We compare the evolved cubic spline ANNs with evolved sigmoid ANNs on synthetic classification problems and a time series prediction task so as to assess the benefits of problem-adapted AF templates.