Application of Attention Mechanism-Based Dual-Modality SSD in RGB-D Hand Detection

Multimodal gesture recognition is a crucial research area in human-computer interaction. This paper proposes a static gesture multimodal recognition technology based on the Single Shot MultiBox Detector (SSD). Firstly, RGB image data and Depth image data are input into the VGG network to extract features. Then, trained features are concatenated in the fusion process, and the weights of features are adaptively learned with attention mechanisms. Results show that combining the two modalities improves model accuracy compared to using RGB images and Depth images separately. Next, the VGG network is replaced with the MobileNet v1 network as the backbone to make the model faster. The proposed method is tested on the Hand Gesture Dataset. The results indicate that the proposed method is superior to the single-modal gesture recognition SSD network.