Wearable Electronic Glove and Multi-Layer Para- LSTM-CNN Based Method for Sign Language Recognition

Communication between normal people hearing-impaired people is usually realized with the sign language. However, the communication quality is limited by the familiarity of the people with sign language. Wearable sign language recognition systems, especially those based on flexible electronic glove, have attracted much attention due to their efficiency for sign language recognition. However, there are still two problems need to be solved in current research: (1) The size or weight of wearable device is usually inversely proportional to the fineness of the hand movement information captured; (2) both spatial features and temporal features are important for sign language recognition, and current algorithms usually do not achieve sufficient extraction. To address the above problems, a wearable electronic glove with new multi-layer parallel LSTM-CNN (Para-LSTM-CNN) network is proposed. The designed wearable electronic glove only 126 g in weigh, and utilizes inertial measurement units (IMUs) and Flex sensors to capture hand motion signals. 41-dimensions of hand motion signals can be captured and upload to computer via Wi-Fi. A Para-LSTM-CNN algorithm based on parallel features extraction strategy is proposed. The temporal and spatial features of the sensor signals are extracted through LSTM and CNN, respectively. The Para-LSTM-CNN can effectively improve the accuracy of sign language recognition by fusing spatial and temporal features synchronously. The proposed system has a high sign language recognition accuracy of 97.01% for 26 gestures. Result shows that the proposed system promises to provide effective solutions for patients with earing-impaired. The available code of Para-LSTM-CNN can be found at https://github.com/1104162390-A/Para-LSTM-CNN/tree/main.