Flexible Activation Bag: Learning Activation Functions in Autoencoder Networks

An active area of research in the field of Machine Learning is the optimization of network structures, including activation functions. However, selecting a suitable activation function is not that simple and usually requires an elaborate empirical process. Unfortunately, this process is very resource intensive and does not always lead to the best solution. This paper presents a novel approach to select, compose and generate adaptive activation functions without any supervision. The best-suited activation function is learned from a collection of function candidates while training the neural network. Standard activation functions are replaced by a sub-network that outputs an indi-vidually parameterized function. The work raises the question whether modern activation functions that are rectified linear unit shaped are beneficial in unsupervised models. We evaluate our approach in autoencoder structures on industrial data and show performance improvements compared to existing methods. The approach is generally applicable to any task and data domain. Finally, ablation studies on the corresponding parameters under-line the cardinality of the approach and performance impacts.