Multimodal Sign Language and Big Data Integration Research

Sign language is the primary means of communication for the deaf and mute community. The regional and national differences in sign language have led to barriers between different sign language systems, making communication between deaf and mute individuals from different regions challenging. This paper proposes a vision-tactile fusion approach to recognize sign language. The scheme applies a flexible sensor data glove to monitor the angle changes of finger joints, as well as visual image processing technology, to integrate tactile and visual data for sign language recognition. The integration is performed through three methods: weighted average, weighted multiplication, and attention weights, with accuracy rates of 95.55%, 97.87%, and 98.83%, respectively.