Understanding Effects of Architecture Design to Invariance and Complexity in Deep Neural Networks

This article studies the role of architecture design, i.e. choice of the number of nodes at each hidden layer, in deep neural networks (DNNs). We give a theoretical explanation that invariance and complexity of a DNN are determined by the design of its architecture. To be more specific, for DNNs with the rectified linear activation function, we prove that the variations of gradients become the largest when the bottleneck layer, the layer with the fewest nodes, changes its activation pattern and the gap of the variations of gradients of the bottleneck layer to those of the other layers is getting larger as the differences of node counts between the bottleneck layer and the other layers become larger. That is, most large variations of the function made by a DNN are concentrated on the decision boundaries corresponding to the activation patterns of the bottleneck layer. An important implication is that invariance and complexity can be improved simultaneously, which is a key to success of DNNs for many artificial intelligence domains, by designing the highest layer as the bottleneck layer. Various experiments are carried out to validate our theoretical result and its by-products.