On the performance of pairings of activation and loss functions in neural networks

The selection of parameters is one of the most important tasks in the training of a neural network. The choice of activation and loss functions is particularly relevant as the formulation of training procedures strongly depends on the pairing of these functions. However, the very few works on the effect of different combinations of these functions do not present a comprehensive experimental study on classification and only investigate a few pairings. This paper provides a significant empirical analysis on the selection of such pairings. Our work presents the formulations of Iterative Reweighted Least Squares for nine pairings of most common activation and loss functions. We investigated the impact of these formulations, including natural pairings, on both binary and multi-class classification in artificial and real-world datasets. Our results show that, for multi-class classification, one should select an activation and a loss function that form a natural pairing in order to generate an effective weight update rule. For binary classification, although different pairings produced statistically different mean average precisions, natural pairings were not a significant factor in the generalisation performance of a neural network.