Should backpropagation be replaced by more effective optimization algorithms?

The authors propose the use of backpropagation (BP) as the preferred technique of optimizing the values of the weights in an artificial neural network. They compare functional representation via BP and a successive quadratic programming code, with the latter being at least four times faster in achieving the same error tolerance. The proposed strategy has two main features. One is that it forgets about adjusting the weights sequentially from the output layer to the input layer, and instead adjusts the entire set of weights at once. The second feature is that it passes the entire set of patterns through the network on one stage of iteration and uses the sum of the squares of all of the errors for all the patterns as the objective function. Another feature of the strategy is that it uses a nonlinear optimization code that accommodates constraints, such as the generalized reduced gradient method or successive quadratic programming, to adjust all the weights and other parameters.<
>