Evaluating Deep Neural Network Performance on Edge Accelerators: A Roofline Model Adopted Benchmarking Approach

This research work aims to run most popular DNNs and performing roofline analysis on the three most popular hardware accelerator units including Google Edge TPU, Intel Neural Compute Stick, and Xilinx ZCU102 FPGA board. The roofline model allows visualization of the selected DNNs and their variant’s performance with respect to the limited computational and memory resources of the investigated hardware unit. In this work, DNN models of Mobilenet, Efficientnet, Alexnet, Regnet, Densenet, Vgg, Resnet and corresponding topologies are investigated for its performance on the aforementioned three hardware units. All the model and its variants suffer from throughput on the neural compute stick, with Resnet showing the best compute-memory balance and highest throughput, whereas Mobilenet suffers from limited memory bandwidth leading to poor throughput. In the Edge TPU accelerator, all the models suffer from limited memory bandwidth and associated data transactions of the USB interface, with EfficientNet model showing the maximum throughput and best compute-memory balance among all the models. Resnet models are considered optimal to run on ZCU102 FPGA board when configured as DPU unit. As expected, noticeable throughput gain for the DNN runs on three-threaded execution over single-thread on the FPGA board was also characterized.