Multimodal Features Fusion for Driver Fatigue Detection Based on CNN-GTN Learning

Driver fatigue detection is one of the crucial methods to ensure driving safety. This paper proposes a driver fatigue detection model through multimodal fusion based on convolutional neural networks (CNN) and gated transformer networks (GTN). Specifically, the driver’s facial features are extracted using CNN and combined with the driver’s electrocardiogram (ECG) and vehicle state features to form multivariate time series. Then, the GTN employs self-attention and masking mechanisms to capture the channel-wise and step-wise correlations of the input, fully leveraging the advantages of complementarity among different modalities. To verify the validity of the proposed model, extensive experiments are conducted on our dataset collected from 15 participants using a driving simulator. Experimental results demonstrate that the fusion of three modalities achieves higher accuracy than a single modality. The trimodal model achieved an average prediction accuracy of 90% across the four fatigue levels. Therefore, the multimodal-based driver fatigue detection method is feasible for autonomous vehicles (AVs).