Weight update in back-propagation neural networks: the role of activation functions

The speed of the learning phase in a classic back-propagation neural network depends both on learning rates and on the choice of activation mappers. These relationships, implicit in the Hebbian learning rule, are analytically analyzed, focusing on the generalized delta rule. A theorem sets a maximum for the step to be taken along the gradient descent direction according to the chosen activation function and to the learning rate. These results explain different requirements for learning parameters, for the hardware representation of weights, and for the behavior of activation function features with respect to learning speed. Results, in order to obtain a significant generalization, are applied to a large activation functions family comprising the most common activation mappers.<>