A Comparison of 2D and 3D Convolutional Neural Networks for Hand Gesture Recognition from RGB-D Data

Hand gesture recognition from videos has more challenges compared with still images due to the difficulty of representing temporal features and longer training times especially in real-time applications. In this paper, we investigated the potential of 2D over 3D CNN's for representing temporal features and classification of hand gestures in videos. We mapped the frame sequence of hand gestures to a chronological tiled pattern in order to capture the dynamics of the hand movement in a single frame. Then, using 2D CNN's, we generated feature vectors containing both special and temporal features. Additionally, we proposed a new approach for fusing data and predictions through a two-stream architecture to exploit depth information. The effects of different types of augmentation techniques is also investigated. Our results confirm that appropriate usage of 2D CNNs outperforms a 3D CNN implementation in terms of recall, accuracy and time in this task.