Multimodal Emotion Recognition with Explainable AI for Cognitive Human-Computer Interaction in Smart Environments

The increasing complexity of human-computer interaction in smart environments has necessitated the development of systems that can accurately recognize and respond to human emotions. Traditional emotion recognition models relying on a single modality—whether text, audio, or image—often suffer from limited robustness, sensitivity to noise, and an inability to generalize across diverse real-world conditions. Existing multimodal approaches attempt to integrate information from different sources but are frequently hampered by inefficient feature fusion strategies, high computational overhead, delayed response times, and a critical lack of explainability in decision-making processes. To address these challenges, this research proposes a novel Multimodal Emotion Recognition framework enhanced with Explainable AI (XAI) to achieve accurate, transparent, and efficient emotion detection in cognitive human-computer interaction environments. The proposed system employs dedicated deep encoders for each modality, followed by an attention-guided feature fusion mechanism that dynamically prioritizes the most informative features. This approach enhances user trust while maintaining operational excellence. Extensive experiments conducted using benchmark datasets demonstrate that the proposed framework achieves an overall accuracy of 89%, significantly outperforming unimodal baselines and traditional multimodal fusion techniques. The system reduces real-time response latency by 70% compared to conventional methods and improves fault recovery and throughput by over 35%, while maintaining a moderate model complexity suitable for real-time deployment. Explainability evaluations further confirm that over 92% of the model's decision pathways are human- interpretable.