Sign Language Detection with Action Recognition Using Temporal Convolutional Networks and Transformer Encoders

Speech disabilities pose significant communication challenges, often leaving people dependent on sign language, which uses hand gestures, body movements, and facial expressions to convey information. However, sign language is not universally understood, leading to communication gaps and social barriers. This paper presents an innovative real-time sign language recognition system to address these challenges and enhance accessibility for people with speech and hearing impairments. The proposed system leverages Mediapipe Holistic to extract key points from hand gestures, body movements, and facial expressions. These keypoints are processed using a hybrid architecture that combines Temporal Convolutional Networks (TCNs) for sequential dependency capture and Transformer Encoders for advanced contextual understanding. The system integrates text-to-speech functionality, converting recognized gestures into spoken language to facilitate seamless interaction between sign language users and nonusers. This approach provides an inclusive solution to bridge communication gaps, empowering the deaf and mute community with greater opportunities for social and professional interaction.