Realizing Data Features by Deep Nets

This article considers the power of deep neural networks (deep nets) in realizing data features. Based on refined covering number estimates, we find that, to realize data features such as the locality, rotation invariance, and manifold structure, deep nets essentially improve the performances of shallow neural networks (shallow nets) without requiring additional capacity costs. Conversely, to realize some data features, such as the smoothness, we show that deep nets perform similar as shallow nets, provided the depth is not extremely large. Both sides show the advantages and limitations of deep nets in realizing data features and demonstrate that deep nets are not always better than shallow nets.