The impact of activation functions on training and performance of a deep neural network

Artificial Neural Networks (ANN) are modelled on the workings of the human brain where information is transported through the synapses formed between individual neurons. The activation function decides the signal to be transmitted from a node of the ANN to the others that are connected, so we can interpret the value of the function as the "activation" of a neuron. There are many activation functions available, each with its advantages and disadvantages when it comes to learning rate and computational load. The current article presents the most commonly used activation functions and their applications in ANNs. We study the effect of using different activation functions for a convolutional neural network used for image classification. Model performance for the train and test set are presented for each fold of the k-fold cross-validation, as well as accuracy scores for the trained network.