A novel neural network training technique based on a multi-algorithm constrained optimization strategy

A novel methodology for efficient offline training of multilayer perceptrons (MLPs) is presented. The training is formulated as an optimization problem subject to box-constraints for the weights, so as to enhance the network's generalization capability. An optimization strategy is used combining variable metric, conjugate gradient and no-derivative pattern search methods that renders the training process robust and efficient. The superiority of this approach, over Off-line Backpropagation algorithm, the RPROP training procedure as well as over the stand alone algorithms involved in the proposed complex optimization strategy, is demonstrated by direct application to two real world benchmarks and the parity-4 problem. These problems have been obtained from a standard collection of such benchmarks and special care has been taken on the statistical significance of the results by organizing the experimental study so as to compare the averages and variances of the training and generalization performance of the algorithms involved.