Emotion Recognition in the Wild: Integrating Multimodal Data for Realistic Insights

Emotion recognition plays an essential role in interactions between human and computers. Multimodal methods using both visual and auditory inputs have surpassed unimodal approaches. The study proposes a multimodal emotion recognition(MER) system that uses visual frames and audio signals derived from video clips in the IEMOCAP dataset. The model uses DenseNet to extract visual features and SoundNet to extract audio features. Emotion classification is performed using fused features and a 1D CNN. In contrast to typical emotion recognition systems, this approach highlights the significance of effective integration of features in incorporating favourable data obtained from the two modes. By integrating visual and audio features, model efficiently collects complex emotional signals that unimodal systems may miss, thereby providing an accurate representation of human emotions. Studies show significant gains in recognition accuracy over unimodal systems. The results show the proposed method’s endurance and flexibility, laying opportunities for its use in real-life situations like emotion-based AI assistants, and emotive systems for computing.