An FPGA-based Quantization and Acceleration Framework for Multi-DNN

Deep neural networks occupy an irreplaceable place in both modern consumer and industrial sectors. The advent of INFerence-as-a-Service (INFasS) by cloud providers facilitates end-users in various domains. Numerous studies have tailored accelerators to boost the efficiency of multi-DNN inference towards these scenarios. However, current research seldom employs quantization techniques to improve the performance of multi-DNN accelerator. FPGAs are characterized by high parallelism and flexible reconfigurability, with their heterogeneous resources enabling support of diverse precision levels. In this context, we propose a framework for quantizing and accelerating multi-DNN on FPGA, including 1. a hardware-aware inter-layer mixed-precision quantization algorithm tailored for multi-DNN inference, 2. an FPGA architecture that facilitates ultra-low bit-width mixed-precision quantization, and 3. a low-overhead scheduling algorithm. Our co-design improves the throughput by up to 2.19x and reduces the response time by up to 28% compared to a unified-precision baseline. In addition, our design achieves similar DSP efficiency and better energy efficiency compared to related work.