Real-Time Russian Sign Language Recognition

This paper describes our concept for recognizing human hand gestures, including sign language of the deaf, using Russian sign language as an example. The key problems that hinder the creation of a gesture recognition system are highlighted, namely: the difficulty of collecting a sufficient amount of data for training classical neural networks, the variability of movements between gestures in continuous signed speech, as well as the limited resources, both computing and hardware, for widespread use of the system. An architecture is proposed that partially solves the identified problems by gamifying data collection through the community of deaf people and optimizing performance by changing the number of significant parameters and distributing the computational load between system elements. We propose a recognition system structure consisting of a detector of the position and configuration of a person's hands, body and face using MediaPipe and step-by-step gesture classification based on data received from the detector. Step-by-step classification is used to increase accuracy with a low amount of training data and simplify the subsequent development of the system due to less retraining of models. The choice of architecture for the sign language recognition system is based on our experience testing previous prototypes. The results presented in this paper demonstrate the potential of the system being developed, but our prototypes are not the final product.