American Sign Language Alphabet Recognition with YOLOv5 Enhanced by MediaPipe Hands

Sign language recognition (SLR) aims to break the communication barrier between hearing-impaired individuals and others, and is beneficial in building an inclusive and caring society. This paper aims to achieve real-time static sign language interpretation from images employing deep learning methods. Conventionally, a pipeline approach is used to locate the position of the hand gesture in an image and then classify the gesture. You Only Look Once version 5 (YOLOv5) is a suitable model that captures the hand gesture with bounding boxes and predicts the gesture with a convolution-based classifier. However, in practice, SLR performance may be significantly affected when gestures are partially occluded, or the background environment is complicated. Therefore, this paper proposes a method using MediaPipe Hands to enhance hand features, enabling YOLOv5 to locate the hand position more precisely. In addition, MediaPipe Hands can predict finger position even when the hand is partly obscured, providing essential features in the classification stage. In experiments, MediaPipe Hands improves the success rate of hand localization in complex environments and increase hand gesture classification accuracy. Compared to the baseline model, the model employing features enhanced by MediaPipe Hands outperformed those without MediaPipe Hands in static sign language recognition (SSLR
¹
). Moreover, our method was tested in real-life scenarios by implementing a web service application and demonstrated improved real-time recognition performance.