Two-Stream Convolutional Neural Networks with Natural Light and Depth Images for Hand Gesture Recognition

Convolutional Neural Networks (CNNs) are applied for hand gesture recognition and achieve desirable performance with homogeneous background scenes, however, there are still challenges for complex human-computer interaction scenes. In this paper, the gesture classification model with Two-Stream Convolutional Neural Networks (Two-Stream CNNs) is proposed. The Two-Stream CNNs structure is built which incorporates both natural light and depth images. And then the fusion features are extracted by convolutional kernels to achieve good performance. Finally, the combination of natural light and depth images is verified to improve the hand gesture recognition accuracy for complex backgrounds. The model is trained and tested on the standard dataset benchmark of American Sign Language (ASL), and experimental results prove that the model can achieve better classification performance on the dataset than the CNNs with single stream. The model improves the classification performance, by 0.9793 F1-score.