A Multimodal Framework for Automated Childhood Reading Disability Screening Integrating Speech, Language, and Eye Tracking

Screening for reading disabilities in childhood is a critical yet challenging task that conventional methods often fail to address with efficiency and accuracy due to a lack of standardization. This study introduces an innovative multimodal framework that synergistically integrates speech, language, and eye-tracking modalities to automate the screening of reading disabilities in children. The groundwork was laid by developing a comprehensive developmental model of Chinese reading abilities through meta-analysis and expert consensus. This model underpinned the creation of a multimodal database comprising speech recordings, eye-tracking trajectories, and behavioral data collected from school-aged children. Employing self-attention mechanisms, our novel end-to-end technique, specifically designed for Chinese reading tasks, exhibited exceptional performance, achieving an F1 score of 93.21% for character recognition and 90.98% for rapid automatized naming in speech assessments. Notably, we devised an embedding representation approach that incorporates domain knowledge to capture intricate multimodal patterns. In conjunction, a cross-modal attention mechanism facilitated effective feature fusion across modalities. Further, ensemble learning strategies combining deep learning with traditional machine learning classifiers yielded an accurate risk prediction rate of 84.21%. A comprehensive evaluation system, quantifying reading abilities through multidimensional eye movement indicators, discerned statistically significant differences between typical readers and those with impairments. Collectively, this framework offers a standardized, efficient, and precise solution for the identification of reading disabilities in children, thereby paving the way for timely interventions and tailored educational strategies.