Text and Voice Integration with Wav2Vec and Specaugment for Improving Accuracy of Speech Emotion Recognition

Emotions are crucial in understanding the human psychological state and can influence social interactions and decision-making. The integration between emotion processing and computational systems enables the creation of more natural and adaptive interfaces. This research focuses on Speech Emotion Recognition (SER), which aims to identify human emotions through voice signal analysis. Many previous studies have been limited to unimodal approaches, and just few have explored multimodal approaches that combine voice and text simultaneously. Furthermore, there is a lack of research that directly integrates Automatic Speech Recognition (ASR) for text as well as voice feature extraction in one step and applies data augmentation to improve model generalization. This research contributes to the improvement of accuracy for emotion recognition tasks using the IEMOCAP dataset with total 5797 voices and text that contain angry, sad, happy, and neutral by incorporating the Wav2Vec model as a multimodal feature extractor (voice and text), and by applying SpecAugment to enrich the data variety. Structurally, our proposed architecture consists of two branches: a voice branch and a text branch. Features extracted from Wav2Vec are sent to the voice branch using an ECAPA-TDNN model, and to the text branch using a BERT model. These two branches are then combined using fully connected layers for final classification. Experiments show that this multimodal approach can achieve high performance, namely weighted accuracy of 90.28% and unweighted accuracy of 90.62%, with requiring special fine-tuning of the text model. These results indicate that the integration of multimodal with pretrained and data augmentation approaches can significantly improve the performance of SER systems.