Sign Language-to-Emotion-Specific Text Translation: A Review

Communication barriers significantly impact the deaf and mute community, who rely on Sign Language—a system of visual gestures and facial expressions—to interact. However, the lack of accessible translation tools hinders their full social participation. Existing solutions primarily focus on converting text or audio into sign language or translating fingerspelling, often neglecting the emotional context of the communication. This project proposes an AI-based system to address this gap by translating sign language into text and detecting and translating the text specific to the signer’s emotions. The system will utilize Deep Learning and Computer Vision techniques, specifically MediaPipe for real-time hand and pose tracking, Temporal Convolutional networks (TCNs) for Sign Language Recognition(SLR) which captures temporal dynamics of gestures and T5(text-to-text transfer transformer) for Sign language translation(SLT) of the text that is specific to an emotion. The dataset will include videos of sign language in American Sign Language (ASL) focusing on everyday vocabulary. The outcome is to enhance communication for the deaf and mute by providing real-time, context-aware translations, including emotional nuances. It would contribute to greater inclusivity and empowerment, ensuring that the voices and emotions of the deaf and mute population are accurately represented and understood in various social interactions.