Speech and Facial Emotion Recognition using Convolutional Neural Network and Random Forest: A Multimodal Analysis

Emotion recognition is a key component of affective computing, which aims to create systems that can understand and adapt to human emotions. One of the challenges of emotion recognition is to capture the subtle and complex cues that humans use to express their feelings, such as facial gestures and voice. This paper presents a multimodal approach to identifying emotions from these two sources, which is useful for many domains such as sentiment analysis, human-computer interaction, and affective computing. Two independent models are employed: one for speech emotion recognition and another for facial emotion recognition. The results of the two models are then combined using decision-level fusion to give the final emotion prediction. The proposed speech emotion model utilizes MFCC features from the IEMOCAP dataset, while facial emotion recognition employs facial coordinates from the same dataset. The speech emotion model employs a network extracting spectral and temporal information through parallel convolutional operations, achieving a WA of 71.36%. A random forest classifier has been used for facial emotion recognition after preprocessing the facial coordinates. This model achieves a WA of 49.31%. After decision level fusion we obtain WA of 72.9%, demonstrating the superiority of the multimodal approach over the unimodal approach.