Investigating Physiological and Behavioral Sensing Modalities Toward Drowsiness Detection

[] Monitoring driver drowsiness is a crucial aspect of ensuring road safety. Many studies have explored a variety of physiological signals and behavioral monitoring of drivers using video, or a combination of these approaches. In this article, we investigate and optimize the effectiveness of various modalities to monitor drowsiness. We developed a physiological model using electrocenography (EEG), electromyography (EMG), electrooculography (EOG), and electrocardiography recordings (ECG). A video-based behavioral model was then developed, utilizing a pretrained ResNet-101, face landmarks and handcrafted features for feature extraction, followed by classification with two long short-term memory (LSTM) blocks. We also investigated a combination of these models using decision fusion to form a hybrid model. The proposed method was trained and evaluated on the publicly available database (DROZY) and compared to other methods on the same database. Our proposed physiological and behavioral models were separately compared with previous approaches where we demonstrated their superior performance when appropriately validated. We further improved the results by combining the physiological and behavioral models which detected drowsiness in 93.10% of trials through leave-one-subject-out cross-validation.