Modal Fusion Sign Language Recognition Based on Swin Transformer Architecture

Sign language plays an important role in information transmission and emotional communication between the hearing-impaired and the outside world. It is expressed through a rapid and complex combination of gestures, body postures and facial expressions. In this paper, Google's MediaPipe framework is used to extract and process key point information of hands. Depth features are used to supplement the RBG diagram. A framework of skeleton sensing multi-modal recognition suitable for complex scenes is proposed. This can be used to deal with the problem of sign language recognition of multi-modal information in different scenes. However, most of the existing sign language recognition methods are traditional convolutional networks and LSTM. Although they have excellent achievements in image recognition and video recognition, their application in sign language recognition does not conform to the characteristics of sign language. In this paper, a Swin Transformer model is used to process both depth information and RGB information. Finally, statistical data fusion method was used to fuse the depth prediction results with RGB prediction results. Experimental analysis shows that this method performs well in sign language recognition in complex scenes compared with existing methods.