High Throughput and Low Bandwidth Demand: Accelerating CNN Inference Block-by-block on FPGAs

A multitude of accelerators have been designed to accelerate the inference of widely-used Convolutional Neural Networks (CNNs). They can primarily be classified into two architectures: the Overlay architecture, which accelerates layer by layer, and the Dataflow architecture, which accelerates the entire model. Each has its own strengths and weaknesses, and we will propose a novel architecture to capitalize on their strengths and mitigate their weaknesses. Our architecture allows optimization for different target network structures. Internally, it features multiple PE Arrays, and during runtime, it can flexibly switch between different interconnection modes among these PE Arrays: the serial execution mode, accelerating an entire block at once, and the parallel execution mode, accelerating a single layer at a time. Its minimum area requirement is close to that of typical Overlay architecture accelerators, while its throughput per unit area far exceeds that of existing state-of-the-art accelerators. When executing the widely-used 8-bit quantized MobileNetV2, we observed an exceptionally high throughput of 2331 FPS (frames per second) on the mid-range FPGA ZU7EV, while requiring only 2.89 GiB/s of off-chip memory bandwidth. Running other networks also demonstrated high efficiency and reasonable off-chip memory bandwidth requirements.