AI-Enabled Sign Language Recognition and Communication System for Speech/Hearing-Impaired Users

The recognition of sign language is of great significance to remove the communication barrier between the speech/hearing impaired and the public. Current intuitive glove solutions are limited to recognize discrete simple gestures (i.e., numbers, letters, or words), far from satisfying the need of the signers' daily conversation. With cutting-edge artificial intelligence (AI), this paper presents a segmentation-enhanced Convolutional Neural Network (CNN) for continuous and complex sign recognition. The proposed sign language recognition and communication system comprising sensing gloves, deep learning block, and virtual reality (VR) communication interface. Both non-segmentation and segmentation assisted deep learning model achieve the recognition of 20 words (93.25% and 82.81%) and 10 sentences (95.5% and 82%) that are picked up as demonstration. To overcome the incapability of recognizing new sentences, the segmentation approach is hence proposed to split entire sentence signal into word fragments, then the deep learning model memorizes all fragments and inversely reconstructs and recognizes sentences. New/never-seen sentences created by new-order word fragment recombination can be recognized with segmentation, largely improving the practicality of the sign language interpretation system. Finally, the AI-integrated VR communication interface allows the bidirectional interaction between signers and non-signers.