A Novel Algorithm for Detecting the Drowsiness Onset in Real-Time

Traffic accidents due to drivers falling asleep while driving is an important cause of death, and effective techniques for coping with this phenomenon are needed. In this paper we present a simple, yet effective algorithm, based on heart rate variability analysis. Results of the experimental validation are reported: two test sessions were performed, one in domestic environments and one in realistic driving environments by using a dynamic driving simulator. The results confirm the results obtained in the previous work (Groppo et al., 2020), and the capability of the proposed algorithm in predicting sleep while driving using a physiological signal that can be collected at the wrist of the driver.


I. INTRODUCTION
TRAFFIC accidents are one of the leading causes of death worldwide. Several studies have proposed that sleepiness on wheel is an important factor in road accidents [1], and most of them pointed out the impact that sleep disorders, such as obstructive apnea (OSAS), have on sleepiness on wheel [2]. Car manufacturers already recognized the importance of preventing sleep while driving, and several approaches have been proposed and deployed; we can categorize them as: • Vehicle-based approach; • Behavioral-based approach; • Physiological-based approach. The vehicle-based approaches are related to the analysis of the steering angle or the rapid acceleration/deceleration [3], [4]. Algorithms have been defined that, based on the above inputs, identify the impending drowsiness of drivers and prompt for actions intended to raise awareness and avoid falling asleep while driving. Behavioral-based approaches are related to non-contact driver monitoring techniques, such as video acquisition and analysis, which compute features such as driver's eyes blinking rate, and/or the percentage of the duration of closed eye [5], [6]. In case the blinking rate or the percentage of the closed eye reaches pre-determined limits, mitigation actions are activated to prevent driving The associate editor coordinating the review of this manuscript and approving it for publication was Nuno Garcia . while sleeping. As far as accuracy is concerned, a recent work [7] reported that vehicle-based approach reaches 63% accuracy (computed as the ratio of true positive + true negative, over true positive + true negative + false positive + false negative), while behavioral-based approaches using eyes blinking rate reach 78%. As far as effectiveness is concerned, with the growing interest in increasing the level of automation of vehicles without reaching full automation (e.g., SAE level 4 or 5), there is a potentially growing number of dangerous driving scenarios where the driver may be falling asleep while the autopilot, not being able to handle the scenario, may require the prompt hand-over to the human driver. In this case, vehicle-based approached are useless in detecting drowsiness conditions, as the driving behavior depends on the autopilot and not on the human driver. Similarly, behavioral-based approaches based on eyes blinking rates may fall short in coping with several scenarios. For example, if the driver wears sunglasses the algorithm may not be able to identify the eyes blinking rate; most importantly, there are scenarios where the driver may fall asleep suddenly with the eyes still open such as in the case of microsleeps and subjects affected by OSAS (Obstructive Sleep Apnea Syndrome). For these reasons, physiological-based approaches to drowsiness detection could offer a significant improvement to the accuracy of driver monitoring systems. Being based on collecting physiological features about the driver, such as the heart rate, respiration rate, etc., the method could promptly VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ identify drowsiness conditions while overcoming the limitations of previously proposed methods. On the one hand, physiological measures are independent from the actions the driver is performing and therefore they can be used in an automated driving scenario where the driver is passive, and we are mostly interested in detecting his/her level of attention, including possible drowsiness conditions. Moreover, physiological measures are not related to eye movements, and therefore they can be used also for detecting drowsiness in subjects that may fall asleep with open eyes and, more, general levels of mental efficiency appropriate to the work to be accomplished. In this work we discuss a physiologicalbased approach that consists in monitoring the Photoplethysmogram (PPG) signal sampled on the wrist of a subject, as introduced in [8]. The PPG signal is processed using a frequency-based analysis to extract a feature we use to predict the drowsiness condition. Extensive experimental results are reported to validate the approach, which can anticipate the actual falling asleep of subjects up to 240 seconds in advance.
Although PPG-based analysis is an established methodology in sleep medicine, the main novelty of this paper lies in using PPG to predict sleep before it takes place. The paper improves the work presented in [19] in two main directions. It adds the calibration part that makes the algorithm adaptable to the wearer physiology, and it introduces the motion artifact removal functions. These two features gives to the presented algorithm greater robustness, which is essential to real-time applications in a realistic environment. Moreover, with respect to other approaches available in literature, the algorithm has been conceived for being deployed in low-cost devices and to run in real-time, thus enabling on-line monitoring of drivers. The rest of the paper is organized as follows. Section II reports a background on human physiology related to the cardiorespiratory system, Section III discusses related works, Section IV presents the drowsiness detection algorithm, Section V presents the experimental validation we performed, while Section VI draws conclusions and outlines future works.

II. BACKGROUND
PPG is an optical measurement technique used to detect changes in blood volume in the microvascular bed of tissue. It is widely used in the clinical field, such as pulse oximeters, vascular diagnostics, and digital beat-to-beat blood pressure measurement systems. The signal is composed of two main components: Alternating Current (AC) and Direct Current (DC). The AC is a pulsatile physiological waveform related to the heartbeat, instead, the DC is the low-frequency components, and it is attributed to the Autonomic Nervous Systems (ANS) activation, the Respiration Rate (RR), and the thermoregulation. [9] Several features can be extrapolated from the PPG. In this work we focus on the Heart Rate Variability (HRV), which measures the activation level of the Autonomous Nervous System (ANS) in its components: the Parasympathetic Nervous System (PSNS, called also rest and digest system) and the Sympathetic Nervous Systems (SNS, identified also as fight or flight response system). The HRV can be calculated in the time domain, analyzing the peak-topeak distance (AC component) of the PPG (the same can be done analyzing the Electrocardiogram, ECG), as well as in the frequency domain by looking at the power spectrum the PPG signal (DC component). [9] In this paper we adopted the frequency domain approach, which computes a variable, related to HRV, as: The nominator LF is the peak of the PPG power spectrum in the frequency range [0.04-0.15) Hz, and it represents the activation level of the SNS. The denominator HF is the peak in the frequency range [0.15-0.40] Hz, and it represents the activation level of both the SNS and the PSNS. The λ value changes over time, as function of the ANS behavior. During the awake state, the activation level of the SNS is high, which translates to a high value of the LF component. Conversely, in drowsy conditions, we can observe an increase of the HF component and a decrease of the LF component [10], [11].

III. RELATED WORKS
Several works have been presented targeting sleep while driving. De Naurois et al. [12] presented a study focused on the analysis of physiological, behavioral, and mechanical measurement. PPG, ECG, and steering angle are measured when a subject drives a vehicle, and then processed by an Artificial Neural Network (ANN), aiming at detecting the drowsiness onset, that is the time when a person transitions from a state of wakefulness to a state of drowsiness. The ground truth is the information from PERCLOS (percentage of time with eyes closed) and from the position of the head. This approach is based on post-processing data after their acquisition and it is mainly focused on the classification of sleepiness. The HRV characteristic is widely used in the biomedical field, Fujiwara K. et al. propose a method capable of predicting epileptic seizures, based on the characteristics extracted from HRV [13]. An interesting methodology was presented by Bianchi A. M., et al., for the detection of the sympathetic-vagal balance control mechanism affecting heart rate applied to subjects suffering from transient ischemic attacks [14]. Another work on a multidomain and multifactorial analysis aimed at detecting somnolence was proposed by Awais et al. [15]. Data was collected from ECG and EEG, from this, HRV, Heart Rate (HR), and statistical measures are extracted after being classified using a Support Vector Machine (SVM based on the wake/sleep/drowsiness classification using HRV extracted from the Electrocardiogram (ECG) and, as ground truth, the sleep/awake score from the EEG. The classification algorithm is based on some statistical operators able to classify the onset of somnolence based on some parameters extrapolated from the ECG representative of HRV, such as the SDNN, the RMSSD, the LF, the HF, etc. [8]. A further work that targets drowsiness onset detection is discussed in [18] where the HRV signal is analyzed to extract features that are indicative of an individual transition from a wakeful state to a drowsy state. The extracted features fed an artificial neural net (ANN) that has been trained using the same features to identify when an individual undergoes the aforementioned transition to drowsiness. The previous works are either aiming at classifying sleep, or at predicting sleep onset using sophisticated approaches such as SVN or ANN. In this work we present a novel approach to identify the drowsiness onset using an algorithm that does not require training, and that can be deployed on a low-cost wearable device.

IV. DROWSINESS ONSET DETECTION ALGORITHM
This section describes the drowsiness detection algorithm we developed. The algorithm processes a PPG signal acquired with a sampling frequency of 50 Hz, that is collected either at the wrist, at the fingertip or at the earlobe of a subject under test.
The algorithm processes the PPG as non-overlapping windows of 2048 samples. The window size has been selected by trading off the computational complexity of the algorithm in terms of memory occupation, with the frequency resolution of the analysis. Let N be the number of 2048-window sampled at a given time. Before executing the algorithm, a calibration phase is performed aiming at identifying the subject condition while awake. This entails observing the PPG signal for the duration of one 2048 samples window, and computing λ, and its prime derivative dλ. Assuming the subject under test is fully awake, the result of the calibration phase is the threshold value dλ T , which is the maximum variation we are observing in that window for the value λ, which is related to the HRV of the subject. Then, for each N 2048 samples window, we compute the Fast Fourier Transform and the Power Spectral Density of the PPG signal. If the PSD is affected by artifacts due to movements, the subject is awake (please note that the artifacts are affecting the PSD only in case the subject is moving, i.e., he/she is walking; they are not affecting the PSD if the subject is still while seated on a vehicle in motion). Upon validating the data set as artifacts free due to movements, we compute the number of windows in which the derivative of λ, that is dλ, is greater than the limit dλ T . In case such a value greater than N/2 indicates the drowsiness onset has been detected and the subject is drowsy, otherwise is awake.

V. EXPERIMENTAL VALIDATION
This section describes the experimental validation we performed to assess the effectiveness of the proposed algorithm.
Two validations were performed. A preliminary assessment during which subjects have been instrumented at home, and a field test using a dynamic driving simulator.
The preliminary assessment focused on the sleep analysis of a certain number of healthy adult subjects (17 subjects, 7 male and 10 female, average age 44.3 yrs., ranging between 18 to 81 years) from the sleep medicine viewpoint. Each subject was instrumented with a state-of-the-art polysomnography device (COMPUMEDICS Somtè PSG4) acquiring the following signals: Academy of Sleep Medicine (AASM). The following states have been scored: • Non-REM 1-2-3 and REM sleep stages • Movements during sleep • Waking state In addition, the waking phase has been further differentiated into active, quiet, and quiet with eyes closed based on the video acquisition, in order to better define the transition phase between waking and sleeping. At the end of the recording analysis, the epochs relating to all the transitions of behavioral status have been reported on a table (expressed in hours, minutes, and seconds). Among them, we have the sleep onset, which is the epoch when the subject started sleeping. Independently from the sleep scoring performed manually by the sleep expert medical doctors, a MATLAB implementation of the algorithm 1 was used to process the PPG signal acquired during the polysomnography, to obtain the epoch of the drowsiness onset. The drowsiness onset epoch is the time corresponding to the 2048 PPG samples window where the algorithm identifies the drowsy state. The obtained results are reported in Table 1, where for each subject we reported the sleep onset epoch and the drowsiness onset epoch in terms of time of day in [hours:minutes:seconds]. As the test is conducted in the evening starting before the subject goes to bed, only the first falling asleep event is reported. Although some subjects experienced short awakening events during the night, they are not reported in Table 1. As the reader can observe, in 16 out of 17 cases the proposed algorithm was able to identify the drowsiness onset in advance with respect to the sleep onset. The algorithm predicts, on average, the falling asleep about 6 minutes and 39 seconds in advance, with the worst-case prediction being 1 minute and 44 seconds before the falling asleep, and the best-case prediction 13 minutes and 37 seconds. In one case, subject ''p'', the drowsiness onset is predicted 4 seconds after the sleep onset, that it later than the actual falling asleep. This result is due to the poor quality of the PPG signal recorded for the subject, for whom the PPG sensor was not properly installed and was affected by significant noise.
Following the preliminary assessment, the second round of experimental activity has been carried out using a dynamic driving simulator (AVL Driving Simulator from Graz) to assess the capability of our algorithm in the most realistic scenario. Subjects were instrumented using the same polysomnography device used during the preliminary evaluation, and they were asked to execute a pre-defined driving mission: the starting point was a city night environment, and then, after 10 minutes of city driving, the car was headed to a highway following an almost straight and regular path. The mission continued on the highway either until the driver fell asleep or for the total driving time (e.g 1 hour). The subjects were sleep-deprived so to maximize the likelihood of falling asleep while driving. After the completion of the driving mission, the PPG channel of the polysomnography has been processed using the PPG-based sleep prediction algorithm 1, obtaining the drowsiness onset epoch reported in Table 2. Concurrently, the sleep expert medical doctor analyzed the polysomnography data, producing the sleep onset epoch reported in Table 2. In this test, 9 subjects participated to the experimentations: 7 of them fell asleep while driving, while 2 remained awake. For all the 7 subjects falling asleep, the algorithm predicted the drowsiness with average anticipation of 9 minutes and 9 seconds, with minimum anticipation of 5 minutes and 14 seconds, and maximum anticipation of 15 minutes and 1 second, thus confirming the effectiveness of the proposed approach. For the remaining two subjects: • Although subject ''g'' did not fall asleep, the algorithm provided a drowsiness onset, thus resulting in a false positive. From the PPG behavior in the time domain, it resulted that the false positive was induced by wrist motion, thus causing an artifact that affected the power spectrum calculation. This problem will be analyzed in the next section, related to the Motion Artifact; • Subject ''i'' did not fall asleep, as both the sleep expert medical doctor and the proposed algorithm reported. As the reader can observe, in a realistic environment the algorithm anticipated the falling asleep of 7 out of 8 subjects (excluding the one not falling asleep), resulting in an accuracy of about 88% (computed as the ratio of true positive + true negative, over true positive + true negative + false positive + false negative), confirming the initial observation   we collected in the preliminary evaluation. For the sake of comparison, experiments carried out in a setting comparable to the one we used and presented in [20], reported accuracy ranging from 63% to 78% depending on the adopted method.

VI. MOTION ARTIFACTS (MA)
In this section the method applied for the MA identification will be presented. As presented in the work ''Drowsiness detection using photoplethysmography signal'', written by Kurian et al., in some cases, the voluntary or involuntary driver movements affect the PPG signal quality for a certain quantity of time. [20] We have developed a tool able to identify the MA thus removing False positives wherever possible. It does not filter the signal but just labels that time window as ''awake'', since if the subject is moving, in a way that leads to a MA, it can be assumed he/she is awake. The MA tool needs a calibration phase requiring the acquisition of 2048 PPG samples, during which the driver should stay motionless. Within this window, the power spectrum is calculated and the maximum value in the band 0-0.4 Hz is stored in the LP a0 variable. The tool compares the LP a0 value with the one calculated in real-time for the observed window, named LP ai . Consequently, we can derive the following labeling process: • LP ai >LP a0 , there is a MA and the subject is AWAKE; • LP ai <=LP a0 , there is no MA and then the prediction algorithm investigates the status of the subject.
In Table 3 is possible to see how the data acquired in the realistic environment processed with this tool change: the only false positive has been turned in a true one. Adding the MA tool, the accuracy grows from 88% to 100%.

VII. CONCLUSION AND FUTURE WORK
The paper presented a simple yet accurate algorithm for processing the PPG signal and to determine the drowsiness onset, on average, about 9 minutes before the driver falls asleep in a realistic driving environment. The algorithm was validated both in a domestic scenario and in a realistic driving scenario; the ground truth (sleep onset epoch) was obtained through a sleep behavioral analysis performed by a sleep expert medical doctor, according to the recommendations of the American Academy of Sleep Medicine. As future work, we are aiming at integrating the algorithm into a custom-built wearable device to measure and process the PPG using a lowcost embedded system.

ACKNOWLEDGMENT
This work has been carried out in cooperation with Sleep Advice Technologies Srl (https://www.satechnology.eu/) as well as with Jmedical Srl (https://jmedical.eu/) who made available sleep expert medical doctors for the duration of the activity. The application experiment on the dynamic vehicle simulator in Graz (AT) has been accomplished in cooperation with AVL (www.avl.com) in the framework of the PRESLEEP project, funded by the European Commission through the grant agreement No 761708.
LUIGI PUGLIESE received the B.Sc. degree in mechanical engineering and the M.Sc. degree in mechatronic engineering from the Politecnico di Torino, in 2017 and 2019, respectively, where he is currently pursuing the Ph.D. degree under the supervision of Prof. Massimo Violante. Since 2020, he has been a Research Assistant with the Politecnico di Torino. His research interests include the development and implementation of algorithms in the human-sensing field.
MASSIMO VIOLANTE (Member, IEEE) received the M.S. and Ph.D. degrees from the Politecnico di Torino, Italy. He is currently an Associate Professor with the Politecnico di Torino. He is very active in technological transfer activities with industries in the automotive and industrial sectors on topics, such as functional safety, model-based design, and embedded system design and validation. He published more than 150 articles in the area of testing and designing reliable embedded systems, and he coauthored two books. His main research interests include the design and validation of embedded system for safety-and mission-critical applications, with emphasis on the use of commercial off-the-shelf components like multicore processors and field programmable gate arrays in automotive, avionic, and space applications. He is scientific responsible of a number of research projects in the area of embedded systems for space, avionic, and automotive applications in collaboration with the European Space Agency, Thales Alenia Space, ITT, CNH Industrial, ELDOR, and Magneti Marelli. He served as the Program Co-Chair and the General Co-Chair for the IEEE Defect and Fault Tolerance in VLSI and Nanotechnology Systems, in 2011 and 2012, and as the Program Chair for the IEEE European Test Symposium, in 2012.
SARA GROPPO received the B.Sc. degree in biomedical engineering from the Politecnico of Torino. Moreover, she supports research and development activities and application experiments within Sleep Advice Technologies, carried out in cooperation with AVL and the Politecnico of Torino within the PRESLEEP and AWAKE H2020 projects funded by the European Commission. Her research interests include nanostructured coating and materials for biomedical applications.