Translation of Sign Language into Nepali Speech Using a Context-Aware Neural Network Framework

In recent years, communication between hearing-impaired individuals and the general public has remained a major challenge, especially in Nepal where assistive technologies are limited. The existing convolutional neural network (CNNs) model for sign language translation is often restricted to static gesture recognition, lacks sentence-level comprehension, and provides a minimal contextual understanding. To overcome these challenges, this research proposes the Translation of Sign Language using a Context-Aware Neural Network Framework (TransSign-Net) designed to translate American Sign Language (ASL) into context-aware Nepali speech. Initially, the ASL dataset comprising 68,400 gesture images was preprocessed using min-max normalization. The system then employs a Multimodal Feature Fusion Network (MFF-Net) that integrates the Red-Green-Blue (RGB) frames, MediaPipe extracted hand, and pose keypoints. These features are passed through a Context-Aware Neural Network (CANN) that has both 3D CNN and Temporal Convolutional Network (TCN) layers to capture spatiotemporal dimensions. A Transformer-based Contextual Language Reasoning Module (CLRM) converts glosses into semantically accurate Nepali sentences. Finally, Google Text-to-Speech (gTTS) synthesizes the output as natural speech, and a PyQt-based Graphical User Interface (GUI) enables seamless real-time interaction. The experimental results show that TransSign-Net model achieves 99.9\% classification accuracy and over 95% sentence-level translation accuracy over the existing CNN model.