Projection pursuit and the solvability condition applied to constructive learning

Single hidden layer neural networks with supervised learning have been successfully applied to approximate unknown functions defined in compact functional spaces. The more advanced results also give rates of convergence, stipulating how many hidden neurons with a given activation function should be used to achieve a specific order of approximation. However, independently of the activation function employed, these connectionist models for function approximation suffer from a severe limitation: all hidden neurons use the same activation function. If each activation function of a hidden neuron is optimally defined for every approximation problem, then better rates of convergence will be achieved. This is exactly the purpose of constructive learning using projection pursuit techniques. Since the training process operates the hidden neurons individually, a pertinent activation function employing automatic smoothing splines can be iteratively developed for each neuron as a function of the learning set. We apply projection pursuit in association with the optimization of the solvability condition, giving rise to a more efficient and accurate computational learning algorithm.