Minimax lower bounds for ridge combinations including neural nets

Estimation of functions of d variables is considered using ridge combinations of the form Σ
_k=1
^m
c
_{1, k}
Φ(Σ
_j=1
^d
c
_{0, j, k}
x
_j
-b
_k
) where the activation function Φ is a function with bounded value and derivative. These include single-hidden layer neural networks, polynomials, and sinusoidal models. From a sample of size n of possibly noisy values at random sites X ϵ B = [-1, 1]
^d
, the minimax mean square error is examined for functions in the closure of the ℓ
₁
hull of ridge functions with activation Φ. It is shown to be of order d/n to a fractional power (when d is of smaller order than n), and to be of order (log d)/n to a fractional power (when d is of larger order than n). Dependence on constraints v
₀
and v
₁
on the ℓ
₁
norms of inner parameter co and outer parameter c
₁
, respectively, is also examined. Also, lower and upper bounds on the fractional power are given. The heart of the analysis is development of information-theoretic packing numbers for these classes of functions.