Multimodal fusion stress detector for enhanced human-robot collaboration in industrial assembly tasks

In the modern manufacturing industry, workers are still required to manually perform complex, repetitive, and physically demanding tasks. Collaborative robotics has emerged to assist human workers, reducing physical strain and monotony, while increasing productivity and safety. Nonetheless, cobots still struggle to match the dexterity of human hands and they lack the ability to understand natural language and interpret human needs, leading to worker frustration and stress. Psychological stress is a critical issue in industrial workplaces, as it affects both workers’ well-being and productivity. This work addresses the issue of operators’ well-being in the context of human-robot collaboration within industrial settings. We propose a multimodal approach to detect psychological stress during an industrial assembly task. Data are collected from 12 participants while performing both autonomous and robot-assisted assembly tasks. Our approach combines physiological signals (ECG, EMG, EDA), facial action units (AUs), and voice features to classify stress levels. The extracted features are combined using a late fusion approach involving the use of a self-attention layer. The results demonstrate the effectiveness of our model in predicting stress levels with a weighted F1-score of 0.81. This research paves the way for the development of more empathetic and human-aware robotic partners, capable of adapting their behavior to improve collaboration and operator well-being.