Deep Learning for Hand Gesture Recognition in Virtual Museum Using Wearable Vision Sensors

Hand gestures facilitate user interaction and immersion in virtual museum applications. These gestures allow users to navigate virtual exhibitions, interact with virtual artifacts, and control virtual environments naturally and intuitively. This study introduces a deep learning-driven approach for hand gesture recognition using wearable vision sensors designed for interactive virtual museum environments. The proposed approach employs an image-based feature extraction strategy that focuses on capturing five partial occupancy areas of the hand. Notably, a deep learning strategy using the bidirectional long short-term memory (Bi-LSTM) model is adopted to construct an effective model for hand gesture identification. The bidirectionality of Bi-LSTM enables it to capture dependencies in both forward and backward directions, providing a more comprehensive understanding of temporal relationships in the data. The bidirectional nature allows the model to better capture the dynamics and complexities of hand motions, leading to improved accuracy and robustness. The performance evaluation includes experiments on publicly available datasets, considering virtual and real museum scenarios. The results highlight the Bi-LSTM-based approach’s superiority by accurately distinguishing various hand gestures. The experimental findings demonstrate that combining the five area ratios and Bi-LSTM classification enables robust recognition of diverse hand gestures and effectively discriminates between similar actions, such as slide left and right classes. Additionally, it shows promising detection performance compared to conventional machine learning models and state-of-the-art (SOTA) methods. The presented approach is promising for enhancing user interaction and immersion in virtual museum experiences.