Deep Learning-Based Emotion Recognition Using Fusion of Multimodal Affective Data From Consumer-Grade Wearable ECG and Speech Sensors

Emotion recognition has emerged as a significant interdisciplinary research topic across various fields, including physiology, neuroscience, and artificial intelligence. This work proposes a deep learning-based emotion recognition model to classify emotions using multimodal affective data, specifically electrocardiogram (ECG) signals and speech from consumer-grade wearable sensors. This system has the potential to monitor and detect anxiety, depression, and intervene in cases of harmful thoughts. Deep learning techniques are leveraged in this study to recognize emotions while individuals watch movies, utilizing data from consumer electronics-based wearable watches. A modified convolutional neural network (CNN) based YOLO (You Only Look Once) algorithm is employed to train the model on the fusion of multimodal affective data, combining ECG and speech. ECG and speech data are collected through a smart ECG measurement watch and a digital voice recorder watch. Our results show an accuracy of 81 % for recognizing emotions. The validation of different emotions classified is performed using spectral centroid (SC) and wavelet transform (WT) methods. The proposed model is compared with several state-of-the-art algorithms, including support vector machine (SVM), k-nearest neighbor (kNN), convolutional neural network (CNN), long short-term memory (LSTM), and YOLOv5 to analyze its performance. This paper investigates the potential of consumer electronics to support a real-time multimodal emotion recognition model, empowering individuals to easily monitor their mental well-being from the comfort of their homes for healthcare applications.