Real-Time Caption Generation for the American Sign Language Using YOLO and LSTM

The lack of captions available for live video events on online platforms makes them inaccessible to the speech and hearing deprived communities. In this study, we examine two models: the pre-trained You Only Look Once (YOLO) framework and a Long Short Term Memory (LSTM) model that uses the Mediapipe framework from Google. The LSTM model is constructed by extracting the landmark features of the subjects face, torso and hands from the video frames and using them to predict the depicted gesture. This sets it apart from existing deep learning methods that use only a Convolutional Neural Network (CNN) framework to extract spatial features. We evaluate the performance and demonstrate the superiority of the proposed model against a LRCN model using our custom dataset. The proposed model offers a valuable approach to making live videos accessible to both viewers and speakers of the disabled community, who are currently depending on human translators. The ability to extract keypoints beforehand further enhances the models capabilities and speed.