Multi-Modal Sensor Fusion for Predicting Sleep and Stress Patterns

Stress and sleep have significant impacts on a human's disease and mental health. With the recent advancements in wearable biosensor technology, the number of research on utilizing these sensors to monitor and evaluate sleep and stress has been growing. However, most existing studies analyze stress and sleep separately and do not consider biological signals and physical activities together. To solve these problems, we proposed “MSFNet”, the multi-modal based multi-sensor fusion model. The proposed model utilizes the 3-axis acceleration, heart rate, activity, and global positioning system (GPS) data collected from multi-sensor. Our model consists of the data preprocessing module that handles multi-sensor data and the neural network module that extracts feature maps from each modality and fuses them to predict the 7 metrics of sleep quality and stress levels. Experimental results showed that the proposed model achieved the average macro F1-score of 0.99 on the training dataset and the weighted sum of macro F1-scores was 5.41 for the 7 metrics on the test dataset.