Time-delay neural networks, volterra series, and rates of approximation

We consider a large family of approximately-finite memory causal time-invariant maps G from an input set S to a set of IR-valued functions, with the members of both sets of functions defined on the nonnegative integers, and we give an upper bound on the error in approximating a G using a two-stage structure consisting of a tapped delay line followed by a static neural network. As an application, information is given concerning the long-standing problem of determining the order of a Volterra-series approximation so that a given quality of approximation can be achieved. We have also obtained a corresponding result for the approximation of not-necessarily-causal input-output maps with inputs and outputs that may depend on more than one variable. These results are of interest, for example, in connection with image processing.