AI Chinese sign language recognition interactive system based on audio-visual integration

Sign Language recognition is an important research direction in the field of computer vision. It is very important to recognize sign language efficiently and accurately to promote the two-way communication between the healthy people and the deaf-mutes so as to help the disabled and other people to integrate into the society harmoniously. There are two types of sign language recognition: Static Sign Language and dynamic sign language. The former can be recognized by image classification network and other technologies, and it is relatively mature for now. At present, the research in this field is not perfect at home and abroad. In this paper, an integrated sign language recognition system based on Yolov5 target detection algorithm combined with LSTM network and OpenPose technology is proposed, eventually the system will be deployed on the raspberry pie to increase portability. In order to improve the accuracy and efficiency of model recognition, this paper adopts data enhancement algorithm and changes the skeleton, detection head and loss function of Yolov5 model, finally, the model recognition accuracy of 98.87% was obtained on the open source Chinese sign language data set recognition test set of China University of Science and Technology. Our experiments show that our improved model has strong dynamic sign language recognition ability, and we hope that this can provide an improved idea for future sign language recognition, especially dynamic sign language recognition.