Design of activation function in speech enhanced deep neural network

The activation functions have an impact on the performance of the neural networks. In the deep neural networks training procedure, the derivative of the standard Relu activation function is zero at the negative semi-axis so that it leads to the inactivation of some neurons, and it is slow for the Tanh activation function in networks training. In order to improve the performance of speech enhancement, we propose a locally linearly controllable PTanh activation function, which is a combination of Tanh, Relu and Taylor's series. Compared to Relu, the PTanh makes the derivative no longer constant to zero when the output value of the neuron is in the negative semi-axis, and the speed of learning is improved greatly. The experimental results show that the PTanh is more adaptable. And the convergence effect of the network and the performance of speech enhancement are improved better.