Multimodal Emotion Recognition Using Computer Vision: A Comprehensive Approach

Real-time understanding and response to human emotions have become critical in today’s connected world of prevalent human-computer interaction. This study presents a novel approach to real-time emotion detection using a camera-based system that analyzes both voice and facial expressions. Our method advances the state-of-the-art by integrating deep learning techniques with parallel models for audio and visual inputs through a weighted fusion approach, a strategy not commonly employed in existing systems. Utilizing datasets such as CK+48 and the Ryerson Audio-Visual Database of Emotional Speech and Song (Ravdess), our system achieves superior performance metrics, including an accuracy of up to 92.92% and an $F 1$ score of $\mathbf{9 2. 9 4 \%}$. Encapsulated within a user-friendly microservice with an intuitive online interface, our solution enables seamless real-time interaction using only a webcam and microphone. Ongoing efforts focus on refining the system’s ability to recognize diverse voices and expressions, aiming to create technology that empathizes with human emotions and enhances interpersonal interactions.