MSCU: Accelerating CNN Inference with Multiple Sizes of Compute Unit on FPGAs

The FPGA-based accelerators have shown great potential in improving the performance of CNN inference. However, the existing FPGA-based approaches suffer from a low Compute Unit (CU) efficiency due to a large amount of invalid computation, thus leading to a great performance degradation. In this paper, we find that no single CU can perform the best across all the convolutional layers (Conv-layers). To this end, we propose multiple sizes of compute unit (MSCU), an approach to accelerating CNN inference on FPGAs. The key idea of MSCU is to select the best combination of CUs via dynamic programming scheduling for each Conv-layer, and then assembles each Conv-layer combination into a computing solution of the given CNN to deploy on FPGAs. The experimental results show that MSCU can achieve 3.36E-03 GOPs/Slice on Xilinx Zynq ZU3EG for performance density which is 4.29x higher than other approaches.