Approximation in Lp(µ) with deep ReLU neural networks

We discuss the expressive power of neural networks which use the non-smooth ReLU activation function ρ(x) = max{0, x} by analyzing the approximation theoretic properties of such networks. The existing results mainly fall into two categories: approximation using ReLU networks with a fixed depth, or using ReLU networks whose depth increases with the approximation accuracy. After reviewing these findings, we show that the results concerning networks with fixed depth - which up to now only consider approximation in L
^p
(λ) for the Lebesgue measure λ - can be generalized to approximation in L
^p
(μ), for any finite Borel measure μ. In particular, the generalized results apply in the usual setting of statistical learning theory, where one is interested in approximation in L
²
(P), with the probability measure P describing the distribution of the data.