Neural networks with periodic and monotonic activation functions: a comparative study in classification problems

This article discusses a number of reasons why the use of nonmonotonic functions as activation functions can lead to a marked improvement in the performance of a neural network. Using a wide range of benchmarks we show that a multilayer feedforward network using sine activation functions (and an appropriate choice of initial parameters) learns much faster than one incorporating sigmoid functions-as much as 150-500 times faster when both types are trained with backpropagation. Learning speed also compares favorably with speeds reported using modified versions of the backpropagation algorithm. In addition, the computational and generalization capacity also increases.