Learning efficiency improvement of back propagation algorithm by error saturation prevention method

Backpropagation algorithm is currently the most widely used learning algorithm in artificial neural networks. With proper selection of feed-forward neural network architecture, it is capable of approximating most problems with high accuracy and generalization ability. However, the slow convergence is a serious problem. As a result, many researchers take effort to improve the learning efficiency of BP algorithm by various enhancements. In the research, we consider that the error saturation (ES) condition which is caused by the use of gradient descent method, will greatly slow down the learning speed of BP algorithm. Thus, in the paper we will analyze the causes of the ES condition in output layer. An error saturation prevention (ESP) function is then proposed to prevent the nodes in output layer from the ES condition. We also apply this method to the nodes in hidden layers to adjust the learning terms. By the proposed method we can not only improve the learning efficiency by the ES condition prevention but also maintain the semantic meaning of the energy function. Finally, some simulations are given to show the workings of our proposed method.