ASLTranscribe: An Integrated Approach for ASL Detection and Sentence Formation Using Transformer and GPT-2 Model

Proposed ASLTranscribe - a novel approach for ASL and translation of such videos using transformer based models and GPT-2 for generation of sentences. We employ the Google Fingerspelling dataset which includes fingerspelling video clips recorded via webcam. We train the transformer model to recognize separate ASL letters and map them to the corresponding word tokens. Training and validation loss is used for assessing the performance of ASLTranscribe. It achieves training loss of 0.4653 and validation loss of 0.5475 which demonstrates that the model has learned and converged well. The recognized word tokens are passed through a GPT-2 model which generates natural complete English sentences. These generated sentences are compared against reference translations using the Bilingual Evaluation Understudy (BLEU) and Word Error Rate (WER) measures. The findings i ndicate t hat A SLTranscribe provides satisfactory performance with BLEU score of 21, and WER score of 19.2.