A Winograd-Convolution-Based Accelerator on FPGA for Real-Time Object Detection with Effective On-Chip Buffer Access Patterns

A convolutional neural network (CNN) based object detector typically requires billions of multiplications, making it difficult to achieve a real-time end-to-end inference on FPGA. To reduce the number of multiplications, Winograd convolution (WinoConv) appears promising in accelerating CNN-based object detectors. However, designing a generic real-time WinoConv-based accelerator on FPGA is challenging since it requires effectively executing various layers, for example, 3×3 and 1 ×1 layers with strides 1 and 2. To address the problem, this paper proposes a WinoConv-based engine on FPGA for real-time object detection with effective line buffer access patterns. Specifically, we propose a simple yet effective dataflow to process tiled input and output tensors with a stride of 2, mitigating an overhead to reorder feature maps between layers. To enable practical use of the engine, we introduce a hardware-friendly post-training quantization scheme, achieving a negligible accuracy drop, i.e., less than 1% in mean average precision (mAP) for a Tiny-YOLOv3 model on the VOC dataset. Our engine is implemented with Verilog HDL and deployed on a Xilinx Zynq ZCU106 FPGA board. The experimental results show that our engine achieves a latency of 5.134ms to inference a custom Tiny-YOLOv3 model, resulting in a throughput of 292.18 GOPS at the clock frequency of 125 MHz on a real-time video stream. The engine utilizes 1053 DSPs and 105 BRAM.