Development of Vision-Based Mobile Application for Filipino Sign Language Recognition Using Machine Learning

In this paper, we present the design and development a mobile application capable of interpreting 105 Filipino Sign Language glosses utilizing the FSL105 dataset for model development and applying machine learning techniques. In particular, the preprocessing uses OpenCV background subtraction to sample important frames from the video input and MediaPipe (i.e., hand and pose landmark) to extract keypoints, which are consequently used as input to a Recurrent Neural Networks (RNNs), including Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), achieving an overall average Top-5 accuracy of around 92% for the best model. Finally, the mobile application was developed using Server-Client approach where the server interprets the sign language captured by the mobile phone and sent back to the app. The application evaluation using System Usability Scale (SUS) achieved an average score of 82.13%, out of 47 participants, where 27 participants have normal hearing while 20 participants from the Filipino deaf community. The assessment showed good acceptability of the developed Filipino Sign Language Recognition Mobile Application.