Real-time Speech Emotion Detection in Employees using Multimodal Emotion Recognition

The real-time speech emotion recognition system presented in this paper integrates both speech signals and facial expressions to assess employee stress and emotional states effectively in workplace environments. Unlike previous studies that primarily focused separately on either speech or facial expression analysis, our work uniquely combines these modalities using advanced deep learning models such as long short-term memory (LSTM) networks for speech processing and advanced convolutional neural networks (ACNNs) for facial feature analysis. This multimodal approach significantly enhances accuracy and reliability, achieving an overall accuracy rate of 96%, thus addressing real-time latency and accuracy challenges inherent in unimodal systems.