Learning and Recovery in the ReLU Model

Rectified linear units, or ReLUs, have become a preferred activation function for artificial neural networks. In this paper we consider two basic learning problems assuming that the underlying data follow a generative model based on a simple network with ReLU activations. The first problem we study corresponds to learning a generative model in the presence of nonlinearity (modeled by the ReLU functions). Given a set of signal vectors yⁱ ∈ R^d, i = 1, 2, . . . , n, we aim to learn the network parameters, i.e., the d × k matrix A, under the model yⁱ = ReLU (Acⁱ + b), where b ∈ R^d is a random bias vector. We show that it is possible to recover the column space of A within an error of O(d) (in Frobenius norm) under certain conditions on the distribution of b. The second problem we consider is that of robust recovery of the signal in the presence of outliers. In this setting, we are interested in recovering the latent vector c from its noisy nonlinear images of the form v = ReLU(Ac) + e + w, where e E Rd denotes the outliers with sparsity s and w E Rd denote the dense but small noise. We show that the LASSO algorithm recovers c E Rk within an ℓ₂-error of O(√((k + s) log d)/d) when A is a random Gaussian matrix.