Primary study on single index artificial neural networks based on penalized regression splines

In this paper, new single index artificial neural network models using penalized regression splines are proposed. The main difference with the standard artificial neural networks is that the output activation functions are not pre-specified in advance but are re-estimated adaptively during the training process by nonparametric smoothers (penalized regression splines in this study) using available data. The estimated activation functions may not necessarily the commonly used activation functions. Very often, the estimated output activation functions may be approximated by generalized hyperbolic tangent functions or simple low-order polynomials. In fact, the problem-tailored functions may be used as the actual output activation functions for usual neural network training and prediction if necessary. Specifically, estimators based on two types of penalized regression splines, e.g., penalized cubic regression splines and thin plate regression splines, are used in this stuydy to estimate the output activation functions. We will use the powerful particle swarm optimization to search for the optimal neural network connection weights. Real-world datasets will be used to compare the performances of the proposed models and the standard artificial neural network models.