Enhancing Speech Emotion Recognition Through Integrated Audio-Video Analysis

Emotion recognition from multimodal data presents challenges such as variations in speech tone, facial expressions, and real-world noise. Most recent systems rely on transformer architectures, which offer high accuracy and generalizability, but are computationally heavy and impractical for real-time or resource-limited applications. This paper introduces a lightweight multimodal emotion recognition system that maintains high accuracy while being efficient. A TIMNET-like model extracts speech features, while a PAttLite-inspired model analyzes facial expressions and scene context. These features are fused using an optimized technique to improve classification performance. The system achieves 92.08% precision on the RAVDESS data set with a compact model size (72 MB for speech transcription and 16 MB for emotion recognition), making it well suited for real-time deployment. Unlike transformer-based approaches that prioritize accuracy at the cost of computational efficiency, this system balances performance and usability. The robustness of the system makes it a great accessibility tool. It offers potential applications in various fields, including market research, advertising, entertainment, and healthcare. The integration of video context, facial expressions, body language, and scene context can improve the accuracy and depth of emotion recognition. Future enhancements include expanding multilingual support, continuous model refinement, and integration with real-time systems for live streaming and video conferencing.