Translation of Sri Lankan Sign Language to Sinhala Text: A Leap Motion Technology-based Approach

Hearing-impaired and inarticulate people interact with each other through visual languages. In Sri Lanka, the standard visual language is Sri Lankan Sign Language (SSL). Even though SSL is mainly based on British Sign Language (BSL), there are numerous unique signs in the SSL dictionary, as visual languages are evolved on what a person sees and how well he/she can express their ideas in the form of facial expressions and body movements. In SSL, there are signs with multiple meanings and also combined signs which express a piece of particular information that does not have a dedicated sign. Due to the significance of such signs, interpretation of SSL into meaningful text has become a research challenge. At the same time, hand gestures play a major part in expressing information in visual languages. Hence, identifying hand gestures through vision-based or sensor-based approaches is very important in sign language interpretation. However, vision-based approaches are susceptible to conditions such as background color, light variations, etc., and also demand more computing power. Moreover, the sensor-based solutions are better in gesture recognition. In this paper, we, therefore, present a sensor-based combined approach using Leap Motion technology with geometric template matching and Natural Language Processing (NLP) to identify and interpret unique signs, signs with multiple meanings, and combined signs in SSL into Sinhala text. The proposed approach shows very promising results for the interpretation of both static and dynamic signs of SSL.