An Integrated Neural Network Model for Classifying Dysphagia Intensity from Facial Expressions

Facial dysphagia intensity classification is an important technique in the fields of human-robot interaction of intelligent meal-assisting robotics. However, traditional single deep neural network models struggle to capture subtle facial changes and sequential information, hindering accurate hierarchical classification of dysphagia intensity levels. This paper presents an integrated neural network model (INNM) which combines the feature extraction abilities of convolutional neural networks (CNN) and the sequential comprehension of bidirectional long short-term memory networks (BiLSTM). The INNM comprises a feature extraction network and a converged CNN-ConvBiLSTM network. In the feature extraction phase, we enhance the YOLOv8s-cls architecture by integrating deformable convolutions to better capture fine spatial details of facial expressions. Furthermore, we incorporate the CloFormer attention mechanism to effectively perceive both local high-frequency and global low-frequency information. In the converged CNN-ConvBiLSTM network, we develop an algorithm that merges a triple-layer hybrid CNN with ConvBiLSTM, utilizing a jumping-connection strategy to bolster the capacity for memorizing long-term and short-term sequences while enhancing spatial information integration. Extensive experiments reveal that the INNM significantly outperforms baseline models and current state-of-the-art methods in dysphagia intensity classification, achieving remarkable accuracy. Ablation studies further validate the effectiveness of each enhancement.