Deep Neural Network Model and FPGA Accelerator Co-Design: Opportunities and Challenges

With an explosive growth of various neural network algorithms, their high performance implementations on hardware platforms, such as GPUs and FPGAs, are becoming critical as well. Compared to widely used GPUs, FPGAs are considered to be harder for design and optimization even with the help of High Level Synthesis (HLS) tools. However, recent studies have shown that FPGAs can outperform GPUs in speed and power/energy efficiency; both factors are important in machine learning applications. In this paper, we will discuss a simultaneous DNN and hardware accelerator co-design method to push the DNN performance on FPGAs. We first summarize existing techniques and results along this direction, and then propose new ideas to further improve DNN development productivity and design quality. Finally we discuss the challenges we would face and propose some potential solutions.