On Expressivity of Height in Neural Networks

In this work, beyond width and depth in a traditional neural network, we augment a neural network with a new dimension called height by inserting links to neurons in the same layer, which gives rise to the notion of height. These links, referred to as intra-layer links, form hierarchical structures that enhance the network’s approximation ability (expressivity) without increasing its parameter count. We show that this 3D architecture (width × depth × height) is more powerful than conventional 2D networks (width × depth) theoretically and empirically. First, given the same number of neurons, a 3D ReLU network (W × K × H) can generate exponentially more pieces in a piecewise linear function than a 2D one (HW × K), i.e., $\mathcal{O}((2^{H} - 1)W)^{K})$ vs $\mathcal{O}((HW)^{K})$. Next, from the perspective of approximation, a 3D network (W × K × W) can achieve higher accuracy in approximating polynomials and reduce errors exponentially compared to a traditional 2D network (W² × K), i.e., $\mathcal{O}(2^{-2W\,K})$ vs $\mathcal{O}(W^{-K})$. Lastly, numerical experiments on 5 synthetic datasets, 15 tabular datasets, and 3 image benchmarks verify that 3D networks can deliver competitive regression and classification performance. Our code has been released for readers’ free download and evaluation.