Emotion Aware Speech Recognition System Using Deep Neural Networks

Emotions play a critical role in conveying context, intent, and meaning in human communication. However, most speech recognition systems lack the capability to detect emotional cues, limiting their effectiveness in applications requiring situationally aware responses. This paper presents an Emotion-Aware Speech Recognition System utilizing deep learning techniques for integrated speech processing and emotion detection. The proposed system employs the TESS dataset for audio feature extraction using Mel-Frequency Cepstral Coefficients (MFCCs), and utilizes Long Short-Term Memory (LSTM) networks to classify input speech into one of seven emotional states: sad, happy, surprised, angry, disgusted, fearful, or neutral. The dataset was partitioned into training (80%) and testing (20%) subsets, achieving a classification accuracy of 95%. Performance evaluation was conducted using precision, recall, F1-score, and a confusion matrix, demonstrating robustness across all emotional categories. The results highlight the potential applications of emotion recognition in domains such as customer service, healthcare, virtual assistants, and education by enhancing conventional speech processing models. This work contributes to the advancement of multimodal emotion recognition, facilitating the development of emotionally intelligent AI systems.