Deep Learning-Based Voice Production System for Deaf and Mute Individuals Combining Lip-reading and Text-to-Speech Technologies

In modern society, deaf and mute individuals face significant challenges in social and daily communication due to their limitations in speech expression. Although traditional assistive tools have made some progress in addressing the hearing impairments of these individuals, there remains a significant lack of support in speech expression. However, with the rapid development of deep learning technology, new opportunities have emerged. We have developed an innovative wearable vocal device that combines lip recognition technology and text-to-speech (TTS) technology based on a deep learning framework to identify messages by analyzing lip movements and morphology, which are then converted into text and subsequently transformed into speech output through TTS. This allows users to communicate orally with others. By modifying the basic unit of the dataset, our model has demonstrated an accuracy rate of $95 \%$. Additionally, this technology can be applied in various scenarios, such as helping patients with speech difficulties communicate with doctors who do not know sign language, or as a mobile lip reading input method, which serves as an alternative to voice input while protecting user privacy. We aim to provide more technological support to the deaf and mute community, promoting an accessible experience in their social and daily interactions.