Optimization of parallel BP implementation: training speed of 1056 MCUPS on the massive

For the backpropagation (BP) implementation on parallel computers, the hybrid approach of the data and the node parallelization techniques has been widely used to utilize the target computers efficiently. However, nothing related to the optimization technique for that hybrid parallelization has been explored yet. In this paper, we discuss the approach theoretically and propose an optimization technique. Experiments were carried out on the recently developed parallel computer CP-PACS. We show that the experimental results agree well with the theoretical predictions. By using the optimization technique, the maximum training speed of 1056 MCUPS (million connections updated per second) has been achieved.