Design and Implementation of an Easy-to-Deploy Energy-Efficient Inference Acceleration System for Multi-Precision Neural Networks

With the increasing penetration of intelligent edge devices, fast and energy-efficient neural network inference has attracted the attention of many researchers. This paper proposes a mixed-precision neural network architecture and low-power circuit implementation. The proposed circuit supports weights in three precisions and achieves hardware usage flexibility through the im2col algorithm. In addition, we integrated the algorithm and hardware control flow with PyTorch to create a complete solution for users to perform quantized NN training and FPGA acceleration deployment within the same framework. We implemented the system on the KV260 FPGA-SoC platform. We then built several applications to validate that the multi-precision quantization approach is practical for real-world applications. Finally, the proposed inference acceleration system (running at 100MHz) achieved 9.1 and 2.1 times better energy efficiency compared to CPU and GPU, respectively.