High Accuracy and Low Latency Mixed Precision Neural Network Acceleration for TinyML Applications on Resource-Constrained FPGAs

Recent advancements in mixed precision quantization algorithms hold significant promise for Tiny Machine Learning (TinyML) applications that face tight resource constraints. However, to leverage the benefits of mixed precision NNs, specialized hardware is required. The reconfigurability of Field Programmable Gate Arrays (FPGAs) allows tensor computations of any precision. In this work, we propose a streaming architecture with layer-specific processing elements to support interlayer mixed precision NN computations on FPGAs. Additionally, we extend the supported quantization scheme to intralayer level by introducing multiply-accumulation (MAC) units that support intralayer dynamic precision quantization. Bit-serial MAC units are chosen to accommodate more PEs on resource-constrained FPGAs. We successfully implemented a mixed quantized NN for the keyword spotting task on a FPGA-only Digilent A7-35T platform with 62% reduction in resource consumption when compared to the previous FPGA-only implementations. Our implementation also achieves 2 times speedup as compared to the state-of-the-art ASIC implementation while meeting the accuracy requirement of 90% set by MLPerf
^®
Tiny Benchmark.