Generalizations of Wearable Device Placements and Sentences in Sign Language Recognition With Transformer-Based Model

Sign language is widely used among deaf. Many existing studies on Sign Language Recognition (SLR) focus on addressing communication barriers between deaf and hearing people. However, current studies face two challenges: the recognition result is highly dependent on the wearable device placements, and the existence of sentences in the training set. To address the challenges, this paper proposed EasyHear – a Transformer-based Chinese SLR system with a generalization approach. For generalization of wearable device placements, an anchor-based signal rotation correction algorithm is proposed to eliminate the impact of variations in wearing positions. In terms of generalization of sentences, a gesture code is constructed to reflect the closeness of gestures after defining an intimate entropy of gestures. Moreover, a semantic code is developed by training a neural network on a large corpus to reflect the intimacy of Chinese Sign Language (CSL) words on unseen sentences in the training set. A Transformer-based model combined with the rotated gesture signals as well as gesture and semantic codes is suggested to improve recognition performance across various wearing positions and sentences. The results show that EasyHear achieved an average word error rate of 21.60% for samples of 712 commonly used CSL sentences.