Heterogeneous Acceleration of Hybrid PSO-QN Algorithm for Neural Network Training

Artificial neural network (ANN) has successfully provided solutions to many practical problems. One of the difficulties in training ANNs is finding the ideal solution to the network weights quickly. This paper designs an implementation of the hybrid particle swarm optimization (PSO) and quasi-Newton (QN) algorithm on CPU-GPU platform using OpenCL to accelerate ANN training. The PSO-QN implementation combines the strength of the PSO algorithm in global search and the advantage of the QN algorithm in fast convergence rate. A configurable parallel line search implementation and an efficient parallel reinitialization implementation are proposed to improve the performance and reduce data transmission. Experiments show the PSO-QN hybrid parallel implementation on CPU-GPU platform can achieve up to 362x and 8.9x acceleration compared with the C++ implementations of PSO and BFGS-QN on CPU, respectively. Compared with the PSO and BFGS-QN parallel implementations, the training loss of the PSO-QN hybrid implementation at given training time is reduced by 15.16% and 3.97%, and the testing error is reduced by 13.66% and 6.86%, respectively.