EarHear: Enabling the Deaf to Hear the World via Smartphone Speakers and Microphones

Sign language plays a vital role in communication and learning for individuals with hearing and speech disabilities, serving as a common language for the deaf. Current state-of-the-art sign language recognition methods primarily rely on computer vision techniques, but they have certain limitations, including susceptibility to light interference and privacy concerns. Ubiquitous acoustic sensing provides new possibilities for sign language recognition, leveraging its high resistance to interference and cost effectiveness. However, existing methods face challenges in achieving satisfactory results due to environmental interference and the complexity of sign language recognition contexts. In this work, we propose EarHear, a robust contactless Chinese Sign Language Recognition and translation system. EarHear adopts a differential-Doppler data preprocessing method to cleverly mitigate the interference caused by the environment. To further identify differences in the morphology, speed, and direction of sign language actions and distinguish similar gestures, we propose the vision transformer for sign language recognition, which is able to model the context dependence of long-range features and output indeterminate long sign language sequences using an attention mechanism. As a result, computational speed and recognition accuracy are improved. Moreover, we explore a large-scale language-model-based sign language translation, which enables sign language recognition results to follow natural language standards, thus realizing a true sense of sign language recognition. The evaluation results based on 15 Chinese sentences show that our system achieves an average recognition rate of 93.38% and a BLEU-1 score of 80.73% for sign language translation, reaching the most advanced level in terms of accuracy and robustness.