Skin Color Segmentation and Levenshtein Distance Recognition of BSL Signs in Video

Sign language automatic recognition is an important research area with open challenges that aims to mitigate the obstacles in the daily lives of people who are deaf or hard of hearing and increase their integration in the predominantly hearing society in which we live. This paper implements, evaluates and discusses strategies for automatic recognition of Brazilian Sign Language (BSL) signs, which ultimately aims to simplify the communication between deaf signing in BSL and listeners who do not know this sign language, accomplished through the processing of digital videos of people communicating in BSL without the use of colored gloves or data gloves and sensors or the requirement of high quality recordings in laboratories with controlled backgrounds or lighting. An approach divided in several stages was developed and all stages of the proposed system can be considered contributions for future works in sign language recognition or those involving image processing, human skin segmentation, object tracking etc. For the skin color based segmentation stage, in particular, several techniques were implemented and compared and the strategy used for sign recognition, exploring the Leveshtein distance and a voting scheme with a binary classifier, is unusual in this area and showed good results. From the original 600 samples of 30 words, chosen for frequency of use and superposition of sign elements to make recognition more complex, the system was able to correctly segment 422 (70%) signs, for which it reached 100% accuracy in recognition using our strategy. This sign database with 600 samples in video of the chosen 30 word vocabulary is another of this work's contributions and is available upon request to the authors.