RPRB-Net: A Sign Language Recognition Based on Multi-dimension Attentions

Sign Language Recognition (SLR) is the process of translating sign language into text or speech, which can also be viewed as a communication and expression tool between deaf individuals and hearing people. SLR using deep neural network has undergone significant development over the last decade. In this study, we design a feature-enhanced SLR technique based on serval spaces. To aid the network learning features from channel and spatial dimensions, we inject various attention mechanisms into the feature extractor backbone. As a result, we named our approach ResNet + Pyramid-Split-Attention + Recurrent-Criss-Cross-Attention + BiLSTM Network (RPRB-Net). Besides, we employ the BiLSTM architecture to encode high-level representations in temporal sequence. The network is ultimately trained using an end-to-end CTC alignment strategy. Extensive trials show that RPRB-Net achieves WERs of 3.5% and 23.6% on the USTC and RWTH-2014 datasets, respectively. Besides, it implies that RPRB-Net significantly surpasses the cutting-edge SLR approaches. Additionally, numerous ablation experiments have confirmed the efficiency of configuration.