Lang2Sign: English to Sign Language Translation Using Generative AI

The accessibility of sign language interpretation is often limited by a shortage and high cost of interpreters. To meet this challenge, we investigate a machine-based approach based on the Conditional Generic Dispute Network (CGAN) [4], particularly the PIX2PIXHD model, to generate sign language gestures from skeletal pose data [2] [13]. Use a media pipe to supply estimates and extract the tactile sensation. This represents body and hand movements as CGAN inputs. This model learns to create fluid body movements and detailed hand gestures that are essential for effective sign language communication. Performance assessments are performed using the loss metrics of the generator and discriminator. This will compare the generated output with the actual sign language gesture. Although models successfully create realistic body movements, recording complex hand gestures remains a challenge.Despite these limitations, our approach illustrates the possibility of a GAN-based model for the generation of SIGN languages. Future research will focus on improving hand gesture solutions, improving model architecture, and expanding data records for better generalization. Improving pose estimation techniques and involvement of additional data sources could further improve the accuracy of the gesture. Furthermore, real-time language recognition can be integrated into the CGAN model to enable real-time translation, making it more accessible to interpreting the interactive language of hearing-impaired people and the listening community [2].Our dataset and baseline deep models are available at https://github.com/AI4Bharat/INCLUDE.