A 65-nm Energy-Efficient Interframe Data Reuse Neural Network Accelerator for Video Applications

An energy-efficient convolutional neural network (CNN) accelerator is proposed for the video application. Previous works exploited the sparsity of differential (Diff) frame activation, but the improvement is limited as many Diff-frame data is small but non-zero. Processing of irregular sparse data also leads to low hardware utilization. To solve these problems, two key innovations are proposed in this article. First, we implement a hybrid-precision inter-frame-reuse architecture which takes advantage of both low bit-width and high sparsity of Diff-frame data. This technology can accelerate 3.2 $\times $ inference speed with no accuracy loss. Second, we design a conv-pattern-aware processing array that achieves the 2.48 $\times $ –14.2 $\times $ PE utilization rate to process sparse data for different convolution kernels. The accelerator chip was implemented in 65-nm CMOS technology. To the best of our knowledge, it is the first silicon-proven CNN accelerator that supports inter-frame data reuse. Attributed to the inter-frame similarity, this video CNN accelerator reaches the minimum energy consumption of 24.7 $\mu \text{J}$ /frame in the MobileNet-slim model, which is 76.3% less than the baseline.

I. Introduction

Deep convolutional neural networks (CNNs) have made a revolutionary breakthrough in the computer vision area. Beside image processing, CNNs are widely used in video tasks, including object tracking [1], [2], video classification [3], and action recognition [4]. CNN models for video [5]–[7] commonly have millions of parameters and billions of operations, which results in long latency time and low battery life on the resource-constrained system.