Speed and Resource Optimization of BFGS Quasi-Newton Implementation on FPGA Using Inexact Line Search Method for Neural Network Training

Quasi-Newton (QN) method is one of the most effective Neural Network (NN) training methods. However, QN training often needs long time especially when the NN architecture is large. The BFGS-QN has been implemented on FPGA for accelerating the training process. The experimental results show that the line search module of BFGS-QN is the most time-consuming module because of its frequent objective function evaluation. In order to solve the issue, an inexact line search method, Armijo-Goldstein (AG) method, is implemented to replace the original exact line search method-Golden Section (GS) method. For the highest training speed, an end-to-end FPGA version of BFGS using AG method is implemented. Moreover, the efficiency AG method makes it possible for hardware-software co-design. The objective function evaluation unit in line search module which consumes the most computational resource is moved to CPU for a speed and resource tradeoff. The experimental results show that the end-to-end FPGA BFGS-AG implementation achieves up to 239 times speed up compared with software implementation. The FPGA+CPU BFGS-AG implementation is up to 153.1 times faster than the end-to-end software implementation and achieves up to 45% LUT, 29% FF and 64% DSP reduction.