m_AutNet–A Framework for Personalized Multimodal Emotion Recognition in Autistic Children

Challenges associated with autism spectrum disorder (ASD) include deficits in interpersonal communication, social interaction skills, and behavior. Autistic children experience difficulties in recognizing emotions and expressing emotions, along with intense emotional upheavals called meltdowns. These outbreaks lead to immense physical and emotional distress in children with autism. Generalized emotion recognition classifiers cannot handle the variations in the prototypical display of affect experienced by ASD children. This paper looks at developing a personalized multimodal neural framework, m_AutNet, that can effectively identify the emotions of autistic children by combining data from their facial and vocal expression modalities. The proposed network includes a personalized facial feature extraction module (that incorporates a distance metric to cluster embeddings with similar labels together and marginalizes dissimilar embeddings), and an audio modality CNN feature extractor that works on speech expression samples of autistic children. Domain adaptation of the multimodal features is achieved through a generative adversarial network tuned with the Wasserstein metric to form a domain-invariant distribution alignment of the feature vectors. A classifier performs emotion classification on this domain space following adaptation. The proposed algorithm shows higher performance than state-of-the-art affect recognition classifiers for autistic children, with an accuracy of 88.25%.