A 2-D Vigilance Estimation Method for High-Speed Rail Drivers With Multimodal Sensors

Accurately estimating high-speed rail (HSR) drivers’ vigilance is crucial for improving operational safety. Existing studies primarily focus on fatigue as the main cause for the decrement in vigilance. However, vigilance is a complex status affected by multiple factors. Therefore, we propose a novel method to evaluate drivers’ vigilance by considering both fatigue and mind-wandering. Firstly, we analyze manifestations of HSR drivers’ vigilance in terms of fatigue and mind-wandering. The results demonstrate that both fatigue and mind-wandering can significantly reduce vigilance, with drivers experiencing mind-wandering exhibiting longer reaction times. Secondly, we construct a 2-D model to evaluate HSR drivers’ vigilance with multi-modal signals. The electroencephalogram (EEG), electrocardiogram (ECG), and eye movement features are fused with a convolutional neural network (CNN), and the fused features are then input into a long short-term memory (LSTM) network to obtain estimation results. To address the issue of imbalanced samples affecting model performance, a Wasserstein deep convolutional generative adversarial network (W-DCGAN) is employed to augment the training data. The average accuracy of our model is 88.75%, with a macro-F1 score of 0.89. The testing accuracies in detecting low vigilance caused by fatigue, mind-wandering, or a combination of both are 82.00%, 85.00%, and 94.00% respectively. These results demonstrate that our proposed method can effectively evaluate HSR drivers’ vigilance from the perspectives of fatigue and mind-wandering.