Japanese Sign Language Recognition based on a Video accompanied by the Finger Images

Sign language is one of the most important tools for hearing impaired people to communicate with each other, and also a good tool to convey messages between hearing impaired and non-impaired people. Most non-impaired people, however, do not know the sign language, and there are many sign languages in the world, which are different depending on countries as with spoken languages. Then, many kinds of research have been performed in the world. Sign language is accompanied by the action using fingers, and the result of our previous work showed that the recognition rate of the words that have the same action with a different number of fingers was low. Therefore, we propose a sign language recognition method based on a video, which is accompanied by the finger images extracted from the overall image. As the result of the experiment using 3D CNN, the average rate of the recognition was 96.9%.