Sign Language Translation System Using Machine Vision

In this paper, we describe the algorithmic solutions we use at various stages of the Russian sign language recognition system algorithm. Our development will improve the socialization of deaf people in society. We describe our software solution for Russian sign language, which is based on the mathematical analysis of data obtained from various existing neural networks and the subsequent transformation of this data into vectors that we use to train our neural network and perform final recognition. Potentially, this development can be adapted to recognize other sign languages. This task is currently especially relevant in the field of machine vision; many organizations and universities around the world are developing in this area. Our approach is unique in that it uses a relatively small array of training data. At the moment, we have learned to recognize 1982 symbols, which is enough for everyday communication of the deaf. In this paper, we briefly described the main innovations in processing continuous video, which allowed us to significantly increase the accuracy and performance compared to previous developments in this area. In this paper, we described video preprocessing, filtering of frames without signs, our algorithm for mitigating outliers in data, and criteria for selecting the most significant features to preserve unambiguity in various external conditions of using our system. At the moment, the accuracy of our system without additional synthetic data generation is close to 75% when recognizing 1982 symbols of Russian sign language, of which almost half are using two hands.