Interactive, Visual-Learning based Tool for Hearing Impaired Children to Improve Language and Cognitive Skills

Hearing impairment is a common condition that affects millions of people worldwide, with approximately 1 in 1000 newborns experiencing some degree of hearing loss. This condition can significantly impact a person's quality of life, causing communication difficulties and social isolation. Early childhood hearing impairment can present significant challenges to a child's cognitive development, making it difficult for them to learn in a traditional educational setting. Consequently, there is an urge for effective learning tools that can assist hearing-impaired children in learning their first language. This paper introduces visual-based and interactive learning tools as a promising approach to enhancing the learning experience and engagement of hearing-impaired children. The proposed system utilizes machine learning to assess a child's initial status and subsequently generates adaptive content while continuously monitoring progress. Moreover, the system incorporates an object exploration feature that enables children to learn from their surroundings. Additionally, it employs natural language learning processes to present contextually similar content, supported by audio and lip movement analysis features that guide correct pronunciation. In response to the ML models utilized, object detection was accomplished through YOLO5, while lip movements were analyzed using LipNet. Regression models were deployed for assessing a child's initial status and subsequent progress. The data collection process involved comprehensive sources that covered a wide array of visual, auditory, and linguistic elements, with the dataset being rigorously tested and verified. Testing implementations were conducted in various real-world settings, encompassing diverse environmental factors and learning conditions. The YOLO model demonstrated an accuracy of 90.3%, LipNet achieved 92.2% accuracy in lip movement analysis, and the multivariate regression models showed a prediction accuracy of 92.8% in evaluating child progress.