A Real-Time Sparsity-Aware 3D-CNN Processor for Mobile Hand Gesture Recognition

A sparsity-aware 3D convolution neural network (CNN) accelerator is proposed for the real-time mobile hand gesture recognition (HGR) system. The complex computation of 3D convolution with the video data makes it difficult for real-time operation, especially in the resource-constrained mobile platform. To facilitate real-time implementation of HGR, this paper proposes two key features: 1) the inter-frame differential aware input method and IDANet to reduce the MAC operation and the external bandwidth by 84 % and 95.2 %, respectively, and achieve 79.97% accuracy on NvGesture dataset; 2) a low-latency 3D-CNN accelerator that utilizes activation and weight sparsity, achieving 31× faster inference latency than the state-of-the-art. The proposed processor is designed in 65 nm CMOS technology. It consumes 262 mW of power and achieves 599 GOPS/W of energy efficiency. As a result, the system realized 1.584 ms inference latency for real-time HGR in a mobile platform.