Multimodal Emotion Recognition Using Heterogeneous Ensemble Techniques

Emotion recognition and sentiment analysis serve several purposes, from analyzing human behavior under specific conditions to the enhancement of customer experience for various services. In this paper, a multimodal approach is used to identify 4 classes of emotions by combining both speech and text features to improve classification accuracy. The methodology involves the implementation of six models for both audio and text domains combined using four different heterogeneous ensemble techniques - hard voting, soft voting, blending and stacking. The effects of each ensemble method on the accuracy for the multimodal classification task are also investigated. The results of this study show that the usage of ensemble learning to combine modalities greatly improves classification, with stacking being the best-performing ensemble technique for the selected collection of models. The proposed model outperforms several existing methods for 4-class emotion detection on the IEMOCAP dataset, obtaining a weighted accuracy of 81.2%.