Maximum Expressive Number of Threshold Neural Networks

Expressive power of neural networks is one of important factors for understanding machine learning. It is wellknown that a deeper neural network also has much higher expressive power than shallower one experimentally, but the difference in expressive powers between activation functions and theoretical explanation have still not been clear. In this paper, we focus on the expressive power of neural networks with threshold activation function and study it by an index of expressive power named "expressive number". Expressive number gives us a type of theoretical limit of learning and enable us to compare expressive powers of different activation functions easily. In this paper, we find the exact value of the maximum expressive number of threshold neural networks, and prove it. Our result suggests deeper neural networks do not always have higher expressive power, and in some cases, rather shallower neural networks have higher expressive power than deeper one.