Detection of Emergency Braking Intention From Soft Braking and Normal Driving Intentions Using EMG Signals

Emergency braking intention detection has a vital practical value for improving driving safety. This paper proposed an electromyography (EMG)-based method to detect emergency braking intention from soft braking and normal driving intentions. Temporal and spectral signatures of EMG signals of emergency braking, soft braking, and normal driving intentions were investigated. Common spatial pattern (CSP) was used to generate virtual channels. The power spectrum density and linear envelope amplitude of EMG signals were used as features, respectively. Chi-square test (Chi) was used to select features. Regularized linear discrimination analysis was developed to detect emergency braking intention from the other two driving intentions. Experiment results showed significant differences in temporal and spectral domains between three kinds of driving. Furthermore, on average, the proposed method based on spectral features can detect emergency braking intention 155.70 ms before behavior under emergency situations with a system accuracy of 95.72%. The proposed method based on EMG signals for predicting emergency braking intention can be applied to develop active driving assistance systems and improve driving safety.


I. INTRODUCTION
Road traffic accidents caused a large number of casualties and economic losses, which becomes a hot topic in today's society. According to the global road safety status report in 2018, road traffic accidents cause an annual death toll of about 1.35 million, accounting for 3% of global GDP [1]. Moreover, traffic accidents are becoming a more important factor leading to death. It is estimated that traffic accidents will become the fifth leading cause of death by 2030 [2]. It is worth noting that the rising death toll is closely related to the number of cars. Under the background of the increasing number of cars, we should pay more attention to the development of assistive driving systems to reduce traffic accidents.
The associate editor coordinating the review of this manuscript and approving it for publication was Xiaojie Su . As a kind of active safety equipment, an advanced driver assistance system (ADAS) can alarm the driver or control the vehicle before accidents [3]- [6]. Driver intention detection, as an important application of ADAS, has become a research hotspot [7], [8]. The signal sources used to detect drivers' intentions can be divided into vehicle information, environmental information, driver action information, and biological signals. The vehicle information includes vehicle speed, tire deflection angle, accelerator, and brake pedal deflection position [9]- [11]. The environmental information includes pedestrian information, road traffic information, and road information [12], [13]. Drivers' action information includes head position, arm position, and foot position [14], [15]. The biological information includes electroencephalogram (EEG), functional magnetic resonance imaging (FMRI), electrocardiogram (ECG), electrooculogram (EOG), and electromyography (EMG) [16]- [20]. By comparing the various signal types, biological signals can be used for early intention detection [21]- [23]. In addition, compared with other biological signals, EMG signals have high reliability and directly correspond to driving related motor action [24], [25]. In this paper, we used the EMG signals to detect driving intention. Several studies have applied EMG signals to detect brake intentions. For example, Haufe et al. [26] set up a car following task in the simulation environment. The front vehicle decelerated from 100 km/h to 60-80 km/h at random during the simulated driving. The driver took emergency braking with the acquisition of EMG signals when the brakelight flashed. Then, they tested their detection model in a real environment [27]. Kim et al. [28] further considered soft braking, and the three driving behaviors were decoded in pairs based on EMG signals, which indicated the feasibility of detecting braking intention based on EMG signals.
However, existing studies do not detect emergency braking intention from soft braking and normal driving and do not analyze temporal and spectral signatures of EMG signals of three driving behaviors. Furthermore, it is worth noting that the braking intention detection performance is low in the existing research. The development prospect of wearable devices based on EMG signals indicates that the detection of braking intention based on EMG signals has great potential application values. For example, it can promote the development of ADAS [29], [30].
In this paper, we aim to explore the temporal and spectral differences in EMG signals of drivers between the three driving behaviors (i.e., normal driving, soft braking, and emergency braking) and proposes a detection method for detecting emergency braking intention from soft braking and normal driving intentions using EMG signals. The contribution of the paper includes two aspects: 1) it explores the signatures in EMG signals of human's lower limb under the three driving behaviors in temporal and spectral domains for the first time; 2) it proposes an EMG-based method for detecting emergency braking intention from soft braking and normal driving intentions, which lays a foundation for developing an active driver assistant system based on driving intention.
The remainder of the paper is organized as follows: Section II describes the methodology. The experiment is described in Section III. Results are presented in Section IV. Discussion and conclusion are in Section V.

A. SYSTEM ARCHITECTURE
The system architecture consists of data acquisition, preprocessing, feature extraction, classification, pseudo-online testing, and performance evaluation shown in Fig. 1. The system first preprocessed the raw EMG signal. Secondly, it analyzed the distinguishability of EMG signals under different braking intentions in temporal and spectral. Thirdly, it calculated temporal and spectral features of the EMG data under different driving intentions. Then, different features of EMG data under different driving intentions were input to a classification system for classification. Finally, pseudoonline testing was used to verify the effectiveness of the proposed system.

B. DATA ACQUISITION AND PREPROCESSING
In the data acquiring part, an amplifier (UEA-32B, SYMTOP, China) was used to collect the EMG data from 8 motionrelated muscles (Rectus femoris, medial femoral muscle, lateral femoral muscle, tibialis anterior muscle, biceps brachii muscle, internal gastrocnemius muscle, external gastrocnemius muscle, and soleus muscle), which are closely related to braking operation [31]. As shown in Fig. 2, the location of the eight muscle channels drawn by the Launch Muscle Premium is illustrated. In the collection system setting, the sampling rate was 1000 Hz, a band-pass filter (0.53-120 Hz) was used to obtain the target frequency band, and a notch filter (50 Hz) was used to deal with the interference caused by the power system [32]. In the processing part of EMG signals, we downsampled the EMG signals to 200 Hz, which was used to reduce the amount of data in order to save the computational time of processing data. Then, baseline correction (BC) and common average reference (CAR) were used to eliminate the baseline drift and the common noises, respectively [33]. Finally, we used a fourth order Butterworth band pass filter to get the frequency range from 15 Hz to 90 Hz [26].
The BC is described as where a i (t) is the ith channel and the tth point of the EMG signals before BC, b i (t) is the ith channel and the tth point of the EMG signals after BC, M is the number of points to calculate the baseline, and here we set it to 20. The CAR is described as where c i (t) is the EMG signals after CAR, N is the total number of EMG channels, and here N is 8.

C. TEMPORAL AND SPECTRAL ANALYSIS
In the temporal analysis of EMG signals, we computed grand-average of the envelop amplitude of braking-related EMG signals segment by linear envelop [32], which rectified and filtered EMG signals with second-order low-pass Butterworth filter with a cut-off frequency of 2Hz. In the spectral analysis of EMG signals, Short Time Fourier Transform (STFT) with 1 s hamming window was used to obtain the power spectrum density (PSD) of braking-related EMG signals segment. Finally, the processed EMG signals segment were superimposed and averaged.

D. FEATURE EXTRACTION AND CLASSIFICATION
The feature extraction and classification part included common spatial pattern (CSP), calculation feature, z-score, chi-squared test, and classifier. First, the CSP was used to generate virtual channels [34]. In this process, we first calculated the covariance matrix of each EMG signals. The covariance matrix is expressed as follows where X (w) is the wth EMG signals (signal points * channels), trace() is trace of matrix, and C is the covariance of X (w). From (3), we obtained the covariance matrix of target samples and non-target samples, and got the mean covariance matrix of target samples and non-target samples. the mixed covariance matrix can be described as where C t is the mean covariance matrix of target samples, C n is the mean covariance matrix of non-target samples, and VOLUME 9, 2021 C m is the mixed covariance matrix. The mixed covariance matrix can be decomposed as follows where F m is the eigenvalue matrix, λ m is the eigenvalue matrix, and the eigenvalues are in descending order. The construction matrix transforms the eigenvalues of the mixed covariance matrix into one, it can be described as where P is the construction matrix, C w is the whiten mixed covariance matrix. From (6) and (7), the matrix transformation can be realized as follows where S t and S n have common eigenvectors, and they can be decomposed as follows where M is the common eigenvectors, and I is the identity matrix. When λ t is larger, λ n is smaller. This property is conducive for classification. The projection matrix of CSP can be described as The EMG samples can be projected into virtual space by W . It can be written as where Y (w) = [y 1 (t), y 2 (t) . . . y m (t)], y i (t) is the ith channel and the tth point of the EMG signals after projection, m is the number of virtual channels. Then, we calculated temporal and spectral features of EMG signals. In the temporal feature of EMG signals, the envelope amplitude of EMG signals was calculated by linear envelope. The temporal features were calculated as follows where T (w) is temporal features matrix, LE() is linear envelope including rectification and second-order Butterworth low-pass filtering with a cut-off frequency of 2Hz.
In the spectral feature of EMG signals, we chose the frequency band ranging from 15 to 90 Hz as our target frequency band in our research. STFT was used to calculate power spectrum density as spectral features of EMG signals. The spectral features were calculated as follows where F(w) is the spectral features matrix, U is the number of points of STFT , and here we set to 200. In order to avoid overfitting, we used z-score to make the data normalization, chi-square test (Chi) was used to select features. The data normalization can be written as where f i (j) is the ith feature of the jth EMG sample, µ is the mean value of the ith feature of all EMG samples, Z i (j) is the ith feature of the jth EMG sample after normalization, and σ was set to one.
The chi-square test can be described as , Zt i (k) represents the ith normalization feature of the kth target EMG sample, Zn i (q) represents the ith normalization feature of the qth non-target EMG sample.
, 1(k) and −1(q) represent the expected value of Zt i (k) and Zn i (q), respectively. χ 2 is the test value, the smaller value means the more consistent with the expected value.
Finally, Regularized Linear Discriminant Analysis (RLDA) was used as classifier. It can be written as where x is the selected features vector, w 0 is the projection vector, and y is the output value. w 0 = (w/µ 1 − µ 2 ), µ 1 and µ 2 are the mean value of target and non-target selected features of EMG sample. w can be obtained by where w is the with-in class scatter matrix, λ is the classification parameter.

E. PSEUDO-ONLINE TESTING
Offline analysis was the first step in detecting emergency braking, which can't be applied to real driving directly. The sample was inputted to the detection model from single trial based on a time sequence in real driving situation. Therefore, further in-depth research was needed. To solve the above problem, pseudo-online can be used. It testing the collected data imitated the way of on-line testing, which set the window width, step size, and input the data of different windows based on time sequence to detection system to obtain the detection results. As shown in Fig.3.

F. PERFORMANCE EVALUATION
In this study, the Area Under Receiver Operating Characteristic (ROC) Curve (AUC) value was used as offline performance metrics [35]. In the pseudo-online testing, the system accuracy rate and advance time were used as performance evaluation metrics. The accuracy rate was the average value of the hit rate and the value of one subtracts false alarm rate, as shown in function (21).
where the Acc is the system accuracy, Hit is the hit rate, and Fal is the false alarm rate. The false alarm rate was the proportion of non-emergency samples judged to be emergency braking. The hit rate was the probability of whether an emergency sample of each trial was judged as emergency braking. The advance time was the emergency braking detection time before pedal deflection. The system accuracy rate was used to indicate the overall performance of the system, and the advance time was used to indicate the effectiveness of the system.

A. SUBJECTS
Twelve subjects with rich driving experience in the driving simulator participated in this experiment. They were all males aging from 24 to 30 years old. The subjects were asked to get enough sleep to keep a clear head and were not allowed to take vigorous physical exercise before the experiment. Meanwhile, the subjects were required to sign the experiment agreement materials, read and sign the experiment synopsis, etc. More importantly, video and transcripts were recorded throughout the experiment. Finally, sixty trials experiments were arranged equally into two different days, and half of the experiments containing emergency braking were randomly distributed as the target trial experiments. The experiment was conducted in accordance with the Declaration of Helsinki. Fig. 4 shows the experiment protocol and virtual, and the experiment was carried out on a simulation platform. The location of the scene was chosen as a reference to urban traffic, and the runway in the scene was set as a straight track with a length of 2600 m. There were three driving behaviors in this experimental paradigm, named x normal driving, y soft braking, and z emergency braking. The specific description of the experiment was as follows: At the beginning of the experiment, the driver was driving in normal driving. The driver looked straight in front of the screen without any unrelated driving behaviors. The driver was kept in the normal driving stage. The speed of the vehicle automatically increased to 120 km/h slowly when the driver drove the vehicle to 1000 m away from the starting point. The driving behavior was switched from x to y, the driver was required to slow down the speed to about 80 km/h when the speed increased to about 120km/h. After finished driving behavior y, the driver released the brake pedal and continued the behavior x. When the vehicle traveled to the road section between 1700 and 2000 m, three pedestrians appeared on the side of the road randomly. One of the pedestrians crossed the road randomly when the vehicle traveled to the position 30 m away from the pedestrians. Once the pedestrian crossed the road, driving behavior was switched from x to z. The driver stepped on the brake pedal urgently to avoid a traffic accident. Then, the driver released the brake pedal and continued to perform driving behavior x until the end of the experiment. Otherwise, the driver kept driving behavior x until the end of the experiment.

C. STRESS ASSESSMENT
After each experiment trial, we asked the subjects to actively feedback on the degree of stress in the experiment. The degree of stress is 1(normal), 2(slightly stressful), 3(stressful), 4(moderately stressful), and 5(very stressful). Table 1 shows the assessment of stress level for each subject. Emergency means the experiment trial with emergency braking, and normal means the experiment without emergency braking. It can be seen that the stress of different subjects in emergency experiment trials is mostly 4 and 5. However, the stress in normal experiment trials is mostly 1 and 2. It effectively shows that all subjects felt stress in the experiment with emergency braking.

B. TEMPORAL AND SPECTRAL SIGNATURE
In the temporal and spectral signatures, the [−4, 4] s interval data of emergency braking and soft braking was selected. However, the [−4,8] s interval was selected in the spectral signatures of soft braking. Note that time 0 s refers to the brake pedal deflection. Fig. 5 shows the temporal signatures of each Channel of emergency braking and soft braking. The amplitude of each Channel changed near the emergency braking and soft braking onset, which indicated that it was possible to distinguish the braking behaviors and normal driving in temporal.
Besides, there were several differences between emergency braking and soft braking. 1. The amplitude fluctuates under emergency braking changed more sharply than soft braking on the whole. 2. Compared with soft braking, the fluctuates time different from the situation under emergency braking.
To sum up, the three driving behaviors can be distinguished in temporal features.     PSD value. The power spectrum amplitude around 50 Hz in the figures was invalid, because of the notch filtering mentioned above. Comparing Fig. 6a with Fig. 6b, the amplitude of the PSD increased near the emergency braking and soft braking onset in the full frequency band, which indicated that it was possible to classify the braking behaviors and normal driving in spectral.
Besides, there were several differences between emergency braking and soft braking. 1. The power spectrum amplitude rose earlier under the emergency braking than soft braking. 2. The power spectrum amplitude under emergency braking changed more dramatic than that of soft braking. 3. The power spectrum amplitude changed under soft braking lasted longer time than that of emergency braking. It should be noted that there were still some differences between the two braking intentions before brake pedal deflection.
To sum up, the three driving behaviors can be classified in spectral features.

C. OFFLINE ANALYSIS OF DETECTING MODEL
In the offline decoding process, a six-fold cross validation method was used to decode emergency braking. We divided 30 emergency braking and 30 non-emergency braking samples into six groups at the same window, and each group included five emergency braking and five non-emergency braking samples, respectively. In each fold, five of six groups were used to train the decoding model, and the rest one was used to test the decoding model.
For the detection of emergency braking from soft braking and normal driving, the decoding emergency braking versus soft braking was adopted to meet the purpose of selecting an appropriate window for soft braking. After the soft braking window was selected, the classification of emergency braking and non-emergency braking (soft braking and normal driving) was carried out. First, the normal driving samples was chosen from 11 s to 4 s before the emergency braking onset randomly, and we selected emergency braking samples and soft braking samples from 2.2 s to −0.2 s before emergency braking and soft braking onset, respectively. Then, [−2.2, 0.2] s was divided into eight windows according to the one-second window size and 0.2 s step size, and the right time point was used to represent the window. Finally, spectral and temporal features were used to decode emergency braking driving, respectively.
In Fig. 7 and Fig. 8, we can see that the CSP and Chi improved the decoding performance effectively in temporal and spectral features. Therefore, CSP and Chi were both used in the following description of different features. Fig. 7 shows the decoding performance of emergency braking versus soft braking. The decoding performance based on spectral features was better than that of temporal features. In addition, the decoding performance based on spectral features and temporal features were increased with time and remained stable at zero position. Therefore, [−1, 0] s was selected as a target window of soft braking for spectral features and temporal features, which was used to decode emergency braking versus non-emergency braking.
The decoding performance of emergency braking versus non-emergency braking is shown in Fig. 8. The decoding performance based on spectral features was better than that of using temporal features. Furthermore, the decoding performance with different features was gradually increasing over time. The decoding performance with spectral VOLUME 9, 2021  features (88.53%) was higher than that of temporal features (86.18%) on the window of [−1, 0] s. Table 2 shows the optimal decoding performance and corresponding windows of different subjects based on different features. For the peak AUC value, there were some difference between spectral and temporal features. The best decoding performance was obtained by using spectral features, with the peak AUC value of 92.82%± 6.64%.

D. PSEUDO-ONLINE ANALYSIS OF DETECTING MODEL
In the pseudo-online testing, 30 trials of experimental data were collected, and six-fold cross validation was adopted. In each fold, 25 trials of experimental data were used to train the detection model, and the rest was used to test the model.
In the pseudo-online detection process. Spectral features were used to construct the detection model with the window size of one second. The minimum step size was judged by the calculation time of the detection system with spectral features (52.80 ms). 60 ms was selected as the target step size of our detection model. The system accuracy with spectral features across all subjects was 95.72%, and the advance time was 155.70 ms shown in Fig. 9.
It should be noted that we can also obtain the time interval between the intention recognition point and the actual behavior according to the command output time point of the ADAS and the time point of brake pedal deflection in real scenarios. Then, we can calculate the braking distance reduced by the ADAS according to the current speed of the vehicle and the time interval, which can be used to evaluate the effect of the time interval.

V. DISCUSSION AND CONCLUSION
In this study, we characterized the braking intentions of EMG signals in temporal and spectral. By comparing the emergency braking and soft braking in different domain signatures, significant difference can be observed in the two braking intentions. Moreover, there were significant differences between the braking onset and before or after the moment in signatures. Therefore, it was feasible to distinguish the three driving intentions based on temporal and spectral features.
On the basis of temporal and spectral signatures, we further performed offline analysis and pseudo-online analysis of the proposed detection method. Offline analysis initially verified the effectiveness of the detection system based on different features, and pseudo-online analysis further verified the applicability of the detection system based on spectral features.
In the offline analysis, the results showed that the decoding performance gradually increased over time. The decoding accuracies based on spectral and temporal features on the window of [−1, 0] s were 88.53% and 86.18%, respectively, which shows that the effectiveness of the decoding model based on different features. Besides, we further analyzed the decoding peak AUC value and corresponding window of different subjects, the decoding peak AUC value of the decoding model built by spectral features and temporal features were 92.82%± 6.64% and 91.25%± 7.46%, respectively. Spectral features showed better results in offline analysis. Therefore, we used spectral features for the pseudo-online testing.
In the pseudo-online analysis, the limitation of the minimum step size was decided by calculating the running time of the detection system based on spectral features, and 60 ms was selected as the optimal step size. On this basis, the average system accuracy (95.72%) of all subjects was used to verify the applicability of the detection system, and advance time showed that the decoding system in detecting emergency braking from soft braking and normal driving can save 155.70 ms compared with brake pedal deflection.
This research can promote the development of an ADAS. The proposed method can be used to detect emergency braking from normal driving and soft braking in practical driving based on EMG signals. When an emergency braking is detected, the ADAS can directly control the vehicle for emergency braking. The application of the ADAS saves the time to take emergency braking, reduces the occurrence of traffic accidents, and improves driving safety.
Compared with the related studies [28] on detecting emergency braking, we could achieve a better detecting performance. In the offline decoding, these features were extracted from the response-locked segment. At the moment of pedal deflection, the decoding performance of emergency braking versus soft braking by using EMG signals based on temporal features was about 69% in [28], and reached 87.95% by using EMG signals based on spectral features in our research. The decoding performance of emergency braking versus normal driving by using EMG signals based on temporal features was 83% in [28], and reached 90.2% by our method. In addition, the result of decoding emergency braking from soft braking and normal driving by using EMG signals based on spectral features was 88.53% in our study. However, one should note VOLUME 9, 2021 that since the simulation environment, acquisition devices, and subjects were different between the mentioned studies and our research, the comparation in the detecting performance is unfair.
There are some limitations in our study. The first limitation is that the simulated driving situations were limited. For the simulation of soft braking and emergency braking in this study, they were induced by increasing the vehicle speed and pedestrian suddenly crossing the road in a short distance before the car, respectively. However, there are many other situations in real life. Different driving situations may induce different responses, which may affect the intensity of EMG signals. Furthermore, in the real vehicle and real driving condition, on the one hand, the vibration caused by vehicles and road conditions may contaminate EMG signals and therefore impair the performance of the detection method. On the other hand, realistic environments can make drivers generate stronger muscle activity, which may improve the performance of the proposed method. Thus, the proposed method should further be evaluated in real driving conditions and artifacts, and noise should be further reduced, and the current work lays a foundation for future work along this direction.
The second limitation is that the performance of detecting emergency braking was not good enough. The average detection performance of the detection system based on spectral features was 95.72% and save 155.70 ms, which was much higher than the random level. However, in real life, there was a high demand for the performance of detecting emergency braking. Once there is an undetected emergency situation, or the brake is delayed, it will increase the incidence of traffic accidents. Therefore, we should pay more attention to improving the performance of detecting driving intentions and save more time. Based on our research, we can try to find more effective features. In addition, we can explore the fusion of other biological signals or artificial intelligence technologies, such as EEG signals or a temporal-spatial deep learning approach [36].
The third limitation is that we did not consider the effects of sitting postures, subjects, and seat positions on the performance. However, there are differences in drivers' posture, height, and sitting positions. To make our research more applicable, the influences of these factors on detecting emergency braking should be investigated. Not only that, the inclusion of more participants in the experiment will enhance the generalization of the results, we will also consider more participants in our experiments to verify our method.
The fourth limitation is that the proposed method requires drivers to wear a data collection device, which will likely make them uncomfortable and unwilling to use it. The recently developed wireless collecting device (e.g., MYO), which does not need to use gel and skin cleaning, may partly address the issue, although its current measurement precision is not as good as traditional measurement systems. More attempts must be made to make the measurement of EMG signals more comfortable and easier to use. Moreover, from the perspective of usability, the small number of channels is preferred but may decrease the accuracy.
In the future, we will focus on solving these limitations, increasing the authenticity of the scene, increasing the stimulation diversity, considering different ways to improve the performance of detecting driving intentions, and conducting studies on the influence of drivers' differences on detecting emergency braking to pave the way for the application of our research. The current and future research in this area will help further improve driver safety.