User-Aware Multilevel Cognitive Workload Estimation From Multimodal Physiological Signals

In this article, we tackle the problem of human cognitive workload estimation from multimodal physiological signals. Allowing computational assistive systems to infer the cognitive states of users in real-time could provide seamless and more fluid human–machine interaction. We introduce WorkNet, a novel end-to-end sequential deep learning model for user-aware cognitive workload classification in a virtual reality driving simulator. Our architecture exploits user-awareness to bridge the performance-deployability gap in cognitive workload estimation between personalized and generalized models. WorkNet uses a dual prediction head, where one recognizes whose user the data corresponds to, while the other predicts the current user's cognitive state. We evaluate WorkNet on a dataset we collected from 20 participants in eight scenarios of different levels of cognitive workload, each induced by either auditory or visual stimuli. WorkNet differentiates with high accuracy among four different cognitive workload levels, independently on the modality of the stimulus. Also, it extracts meaningful features from the input, independently on the modality of the stimulus and on the users. Our analyses show that WorkNet can adapt with limited data, akin to a calibration stage, to perform on a different modality it has been trained on, and on unseen users.