Square Root Based Activation Function in Neural Networks

Activation function plays an important role in the construction of a neural network. Existing methods such as S-type and Rectified-type activation function have many deficiencies in terms of robustness, which always leads to the problem of gradient diffusion and causes the collapse of the neural network. To address these drawbacks, a more robust approach is proposed that activates levels of the neural network by using square root function (SQRT) as the activation function. Utilizing its characteristics of broad safety margin, symmetrical output center and adaptive data range, the method reduces the risk of encountering gradient diffusion for neural networks in training and applications, and decreases the possibility of the neural network collapse. Experiments are carried out to demonstrate the effectiveness and efficiency of the proposed method. It can achieve average 99.6% accuracy in image classification, higher than 94.7% of Tanh function and 68.3% of Relu function under the same conditions. Moreover, our method has low requirements on initialization, fast convergence rate, better stability after convergence, and adaptive global optimal seeking so as to can get rid of the local optimal solution of the neural network.