American Sign Language Translation Using Transfer Learning

Sign language translation (SLT) is the process of recognizing signs, generating glosses, and translating glosses to a spoken language. Our work focuses on translating American Sign Language (ASL) glosses to English sentences. We use five multitask, transfer learning models to perform these translations. Our method outperforms the state-of-the-art model’s approach by 14.58 points in terms of BLEU-4 on the ASLG-PC12 dataset. Furthermore, we assess the complexity and usefulness of ASLG-PC12 for SLT. Our work indicates its synthetic generation does not accurately reflect the complexity of ASL. In turn, we introduce a new dataset, ASLG-EC23, that better reflects the grammatical and topical diversity of ASL. Our method achieves a BLEU-4 score of 40.10 on ASLG-EC23, setting a baseline for future research.