A Study on a HMM-Based State Machine Approach for Lane Changing Behavior Recognition

In recent years, the development of advanced driving assistance systems (ADAS) has grown significantly within the transportation industry to assist drivers for making safe maneuvers. A major component in developing these assistance systems are driving behavior prediction and recognition models. These models aim to infer driving behaviors based on different sources and parameters using complex mathematical models. Machine learning algorithms are being used increasingly to develop these models. In this contribution, two formerly developed trainable models, which are an improved Hidden Markov Model (HMM) and a state machine model, are combined for the recognition of three lane changing behaviors (lane change to the right (LCR), lane keeping (LK), and lane change to the left (LCL). In the improved HMM, a prefilter is implemented on two sets of observation variables (input variables of HMM): one consisting of distances and velocity deviation, while the other consists of time to collision (TTC) variables. To develop an optimal model, thresholds of the prefilter are optimized using a Non-Dominated Sorting Genetic-Algorithm-II. The aim is to investigate if the proposed model is able to produce estimations with high accuracy (ACC), detection rates (DR), and low false alarm rates (FAR). In addition, the performance based on applying the prefilter on the two sets of variables are compared. Comparisons to an individual improved HMM and an ANN-based state machine approach are also addressed. The obtained results show that the application of prefilter on the TTC variables improves the estimation performance. Furthermore, the proposed approach outperforms other approaches.


I. INTRODUCTION
Based on a recent preliminary report by the European Commission related to road accidents in 2021, an estimate of 19800 lives were lost due to traffic accidents in the EU [1]. Focusing on Germany, road accidents have cost around 2562 lives in 2021, according to the German Federal The associate editor coordinating the review of this manuscript and approving it for publication was Alberto Cano .
Statistical Office (Destatis) [2]. While, this is a decrease of 5.8 % from the previous year, it is still a critical problem faced. Further research shows that majority of the deaths in country roads and motorways involve the passenger vehicles, while in built up areas, most victims are pedestrians [1]. A recent report by Destatis also shows that majority of these accidents are caused by human driving behaviors accounting for 71.8 % of the accidents in Germany in 2020 [2]. This is due to the driver's inability to predict the right action when driving in a complex or different environment [2]. The development of Advanced Driving Assistance Systems (ADAS) plays an important role to tackle this issue for improvements to road and vehicle safety. The ADAS provide support and assistance for drivers to maneuver safely in different environments. Some of the well-known ADAS used are the adaptive cruise control (controls speed of the vehicle while maintaining safety distances) and the collision avoidance systems (warn and alert drivers to avoid collision). Driving behavior prediction and recognition models are important elements of ADAS. The incorporation of driving behaviors with ADAS allows early prediction of driving behaviors and dangerous situations. Driving behaviors are considered as individualized, thus classifying and predicting behaviors by an assistance system enables the related drivers to get better supervision while driving [3]. A system that is able to adapt to individual behaviors of different drivers is considered to be ideal not only for increasing safety, but for the improvement of functionality (development of smart vehicles) as well [3]. Machine Learning (ML)-based approaches are increasingly used for the development of prediction and recognition models as these approaches are able to learn from given behaviors to predict similar behaviors or the driver's intentions. Some of the well-known ML approaches used in research contributions are Artificial Neural Network (ANN) [4], Support Vector Machine (SVM) [5], and Hidden Markov Model (HMM) [6]. In [4], an ANN-based model is used to predict lane changing maneuvers based on different situations, while in [5], a SVM model is used to detect lane changing intentions. A lane changing prediction model is developed in [6] based on an improved HMM, which includes the use of a prefilter. Most of these contributions analyze lane changing behaviors because a significant proportion of accidents occur during a lane change [2].
However, several challenges exist related to the development of such models, like defining optimal parameters for optimal estimations. A reason for this is the lack of understanding on selecting relevant input variables and parameters to be optimized due to the black box nature of the ML-based approaches. Two methods are usually applied to solve this problem. The first method combines two or more ML-based approaches to develop the model as in [7], whereby ANN is combined with SVM to predict lane changing behaviors on a highway. The results based on this work show that the combined model produced better estimation results than the individual models. A combination of ANN and HMM to develop a prediction model is realized in [8], whereby the HMM is used to predict different driving behaviors such as emergency steering, normal cornering (ability to handle bends), and straight line driving. Based on the predictions, an ANN model is used to obtain specific steering wheel angles for the different behaviors. The second method is based on feature selection techniques. Features used to describe a particular driving situation play an important role in driving behavior predictions. Thus, methods such as filter and wrapper methods are used in [9] to select the most appropriate features as input variables in the prediction and recognition models. The wrapper method considered in [9] employs the combination of SVM with a recursive feature elimination method to extract features [10], [11]. Different vehicle variables are combined to develop features using a prefilter in [6], as part of HMM for lane changing predictions. On the other hand, deep learning methods have been applied in recent years as in [12], for automatic extractions of temporal and spatial features, eliminating the need for manual extraction.
In this contribution, a trainable model is proposed by combining a state machine-based approach [13] and an improved HMM [6] for the recognition of lane changing behaviors. The lane changing behaviors considered are lane change to the right (LCR), lane keeping (LK), and lane change to the left (LCL). The state machine models the different lane changing behaviors as states, such that it describes the transition from one state to another based on specific conditions. These conditions are defined by the estimations of HMM. A state machine model as a new ML-based model is considered as it is more interpretable than a HMM model, while the HMM is known for its stochastic properties. To improve the performance of HMM, previous works [6] and [14] considered a prefilter application. The prefilter process variables to generate input features. Thus, in this work a prefilter is implemented to two sets of variables as part of the HMM. One set comprises of distance and velocity deviation variables, while the other set comprises of time to collision (TTC) variables. Similar to [6], this work also considers the optimization of the prefilter parameters using Non-Dominated Sorting Genetic-Algorithm-II (NSGA-II). Different from previously mentioned literature, a state machine-based model [13] is combined with HMM to formulate a new ML model for lane changing behavior recognition. The objective is to develop an effective model with improved performance in accuracy (ACC), detection rates (DR), and false alarm rates (FAR). Performance comparisons between the application of prefilter to the two sets of variables are realized as well for evaluations. The aim is also to test the generability of the model such that same parameter values can also be used to obtain optimal estimations for different drivers. In addition, comparisons between the proposed model, an individual improved HMM model [6], and a previously developed ANN-based state machine model [15] are performed to evaluate the effectiveness of the new model. This paper is organized as follows: in Section II the methodology of the state machine model, HMM, and improved HMM are described. Thereafter, the HMM-based state machine model is introduced in Section III. The feature variables used and optimization process of design parameters are described in this section as well. In Section IV, the application of the method is described, which includes the experimental setup, data processing as well as training and test process. The experimental results are discussed in section V. Finally, a conclusion is summarized in section VI. VOLUME 10, 2022

A. STATE MACHINE MODEL
State machines are used to model behaviors using a discrete number of states. A typical state machine model can either remain in the same state or describe the transition to another state based on a set of inputs and conditions. In this contribution, the transition conditions and parameters of the model are defined by the designers. One of the benefits of using a state machine model is its easy designing process and flexibility [16]. Another advantage includes its easy state reachability as the states can be defined finitely [17]. Nevertheless, a state machine-based model as a ML model for behavior estimations has not been widely applied in research. By far, only a few research works have applied this model in various areas, such as in tribology experiments to develop a lifetime model based on acoustic emission data [18] and in driving behavior experiments to develop driving behavior recognition models [13], [15]. Typical state machine models are developed in [18] and [13] whereby, the conditions for state transitions are based on threshold limits of certain variables. For an example, in [13] different threshold conditions of driving variables (values of input variables are higher or lower than the threshold values) define the state transitions.
In this contribution, the state machine-based ML model structure developed in [13] for the recognition of lane changing behaviors is adapted. The three lane changing behaviors estimated are represented by three states. In the developed model, the states are the output of the model at a particular time point. Based on Fig. 1, if the current estimated state is LK, the next possible estimations are either switching to states LCR, LCL (estimating the driver performs a lane change maneuver) or remaining in the same state (estimating the driver remains in the same lane) depending on the transition conditions. On the other hand, if the current estimated state is LCR or LCL, the next possible estimations are remaining in the same state (driver performs further lane changes in the respective direction) or to switch to state LK (the lane change is over).

B. HIDDEN MARKOV MODEL
The HMM is a probabilistic graphical model used to represent behavioral changes. As shown in Fig  To apply the HMM-based model for the recognition of the lane changing behaviors, the model is first trained with the Baum-Welch algorithm (a special case of EM) to estimate λ. Given an observation sequence and the possible hidden states sequence, λ is estimated through learning to best fit both sequences. Then, using the Viterbi algorithm the most probable lane changing behaviors sequence is estimated based on the saved parameters.
The transition probability of switching from S i to S j (a switch from one behavior to another) can be formulated as The probability of being at S i at time t and changing to S j at time t + 1, given the observation sequence is defined as whereby, q t is the state at time t. Hence, the expected number of transition from S i to S j is the the sum of η t for all time steps, while the expected number of transitions from S i is the sum of all transitions from S i . Thus, the a ij is formulated as whereby, T is the time length of the full drive.
Here, χ t (i) is the probability of S i at time t formulated as Thus, b ki can be finally defined as such that the sum of χ t (i) for the entire time length of the drive for a given observation is divided by the sum of Based on the defined a and b, the HMM parameter is generated using the Baum-Welch algorithm. The next driving state is determined based on the saved HMM parameter and the given observation sequence by employing the Viterbi algorithm. Here, the maximum probability of all previous state sequences leading to the next state (q t (j)) is considered to denote the most probable path at time t. So, q t (j) is formulated as The HMM posses certain benefits such as its ability to analyze time series data and its stochastic characteristics [19], [20]. Since upcoming behaviors are stochastic and only depend on the present state, the HMM is a suitable choice for driving behavior estimation. The HMM can also handle temporal pattern recognition [20].

C. IMPROVED HMM WITH PREFILTER
Using a conventional HMM may have an impoverished performance if the features used are not accurate enough. Therefore for performance improvement, various approaches have been established such as a combination of HMM with other methods and HMM-derived methods [21]. The combination of HMM with other methods includes ANN-HMM [22], Fuzzy Logic (FL)-HMM [22], and Gaussian Mixture Model (GMM)-HMM [23]. These methods use results from one method as input to the other to determine the final behavior estimation. The other method is also used for determining the parameters and classifying different driving styles, behaviors, or situations. Thus, the combined methods consider the advantages from both the HMM and other methods to determine the final outcome. On the other hand, the HMM-derived methods such as Hierarchical HMM [24] and Bayesian Nonparametric HMM consider the time series property of HMM [25]. The general idea of HMM-derived methods include partitioning behaviors into several task layers. In these methods, the initial layer is used for determining different driving variables like acceleration, while the higher layer uses the results from the initial layer to estimate the corresponding driving behavior. A new and improved HMM-derived method developed in [6], which includes the application of a prefilter is utilized in this work. The prefilter is applied to the observation variables of HMM.
In the HMM model, the observations variables are dynamic which changes with time. Changes in the observation parameters changes the observation vector. To simplify the model, a prefilter is applied on the data of the observation variables to quantize the variables with a feature vector. The feature vector is used to determine different driving situations [6]. The prefilter divides the driving variables into segments with thresholds, such that each segment represents an observation. Thresholds are defined using optimization to develop the observations and ultimately the observation sequence.

III. HMM-BASED STATE MACHINE MODEL
A new HMM-based state machine model is introduced in this section. Here, two trainable systems, a state machine and an improved HMM are combined to develop a model that recognizes lane changing behaviors. The state machine model describes the transition between the states, while the estimations of an improved HMM define the transition conditions. The transition conditions differ from [13] which uses threshold conditions instead. The structure of this model is similar to the ANN-based state machine model [15], which uses the ANN estimations instead as the transition or remaining conditions. Driving decisions depend on environmental variables as well as individual driving behaviors. Hence, environmental variables describing the relationship between the ego vehicle and surrounding vehicles are selected as inputs.

A. HMM-BASED STATE MACHINE APPROACH
Based on Fig. 3, for a transition from LCR or LCL to LK to occur, the estimation of HMM should also be LK. On the other hand, for a transition from LK to LCR or LCL, the HMM estimation should be LCR or LCL respectively. If the HMM estimation is same as the current state or the conditions are not met, the model remains in the same state. The conditions for state changes are based on the aforementioned HMM mathematical process. The transition conditions are summarized in Table 1.

1) DATA SELECTION
As mentioned previously, only environmental variables are used as inputs for the model as these variables provide information that mainly affect the human driving decisions. Environmental variables are distinguished into two types: state of ego/surrounding vehicles and driver's operational information. Accordingly, two models are developed using two sets of input variables. Model I uses distances, velocity deviation, and current lane as inputs, while model II uses TTC and current lane as inputs. The selected variables based on both models best describe the current driving situation, thus affecting the driving behaviors (given in Table 2 and 3). The variables are selected as they have a higher influence on the driver's decision to make a lane change as well as the ability to describe the relationship between the ego vehicle and surrounding vehicles [3], [6].
A prefilter using two thresholds is applied on the distance and velocity variables for model I and on the TTC variables  for model II. Each variable is divided by the prefilter into three segments [6]. As for the current lane number, the values are fixed indicating the specific lane of the ego vehicle.

B. OPTIMIZATION
The prefilter thresholds are design parameters of the model that needs to be optimized to generate an optimal λ = (A, B, π) for performance improvements. Here, the Non-Dominated Sorting Genetic-Algorithm-II (NSGA-II) is chosen to optimize these parameters. This technique is used due to its ability to handle Multi-objective optimization problems (MOPs) [26] and its fast convergence [26], [27]. The parameters are defined using this approach, such that the objective functions are minimized.
To evaluate the model's performance, ACC, DR, and FAR [28], [29] metrics are used by comparing the actual and estimated behaviors. The metric values are defined based on the True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN) values. Using LCR as an example to illustrate these values, TP is the number of events when the actual and estimated maneuvers are positive (LCR), while FP is the number of events when the estimated maneuver is positive, but the actual maneuver is not. Similar concept applies to TN and FN. The formulations for the metrics are given as Therefore, the parameters are determined such that the model achieves in parallel high ACC, DR, and low FAR. Similar objective functions as in [13] and [15] are chosen for the NSGA-II. As objective functions (13) are used, whereby each function corresponds to a specific lane changing behavior. In Fig. 4, the optimization process of the prefilter thresholds is shown.

IV. APPLICATION OF THE METHOD
The application of the HMM-based state machine approach is realized in this section. The experimental setup for data collection is first described followed by the process of labeling lane changing behaviors. Next, the training and test processes are explained. The training is done using the driving data from each individual driver for defining the optimal design parameter values. Then, the test is done based on the trained model using different data by the same driver and other drivers.

A. DESIGN OF THE EXPERIMENT
The driving experiments are conducted using a driving simulator, SCANeR TM in a laboratory environment (Fig. 5)     t lane (Fig. 6). A LCL is defined by an increase in l, while a LCR is defined by a decrease. When l is the same as the previous time point, a LK is denoted. The beginning of a lane change is defined by the time of the indicator activation, t indicator . Here, the interval between t indicator and t lane is defined as the lane change duration denoted by t change [3], [6]. From the experiments, this duration is between 2 to 3 seconds, whereby the driver activates the indicator 2 to 3 seconds before performing a lane change. Different preset t change values of 2 s, 2.5 s, and 3 s are therefore tested for labeling behaviors to evaluate the impact of t change on the lane change recognition abilities. To do so, the HMM developed in [6] is used to estimate the lane changing behaviors based on the variables of models I and II. Previously mentioned metrics are used to evaluate the estimations. The training and test split ratio is also 70:30. In Table 4, the average performance values based on all drivers using the different t change values are given. Using 2.5 s to label the behaviors, generates results closest to the actual behavior for both models as most metrics have the highest values. Hence, t change of 2.5 s is used to define a lane changing behavior in this research. Inaccurate data are removed as part of the labeling processing. For an example, when the driver does not intend to change lanes, but drives over the white lines or slightly overlaps the lines to the next lane due to driving errors. A lane change is detected consequently, when it does not reflect the actual driver's behavior. Hence, these inaccuracies are removed [6].

C. TRAINING AND TEST
The training and test process are explained here.
Training phase: The purpose of the training is to develop a model with optimal parameters. The training process is described in the following manner, 1) Data (input variables) and the actual lane changing behaviors are loaded into the model. 2) The design parameters (prefilter thresholds) are developed using NSGA-II, which are then used to define the observation sequence in the HMM. Based on the defined observation sequence and actual driving behaviors, the HMM model is trained by optimization to define the optimal HMM parameters. 3) Next, the hidden states are estimated using the HMM parameters. 4) The state machine defines the final estimations using the estimations of HMM from the previous step. 5) The actual and estimated behaviors are compared to evaluate the ACC, DR, and FAR values. Using these values, the objective functions are calculated. 6) Steps (1) to (5) are repeated until convergence, such that optimal parameters are defined minimizing the objective functions. For the number of iterations, a generation size of 200 and population size of 90 is used in NSGA-II. Test phase: The test is performed using data not used in training as mentioned previously. Thus, based on the trained models for each driver, the lane changing behaviors are estimated using test data of the corresponding driver. Estimation performances are then evaluated using the mentioned metrics. To analyze the generability of the model, the trained parameters based on a specific driver are not only tested using the corresponding driver's test data, but also using test data of other drivers.

V. RESULTS
The performance based on the proposed method is presented in this section. Here, the performance based on models I and II are given and compared to analyze the effects of the prefilter on the different variable sets. The results presented are based on using 2.5 s for the lane changing duration, as it generated the best performance values when tested with the proposed approach for both models. This shows that the estimations are closest to the actual behaviors. Performance comparisons between the proposed approach and other  approaches (HMM and ANN-based state machine model) are presented as well to assess the capabilities of the approach.

A. EVALUATION OF RESULTS
The average values of ACC, DR, and FAR based on the test data for both models are given in Table 5 and Fig. 7. In the table, the average values (test data) are the average metric values when each trained model is tested with the corresponding driver's test data.
As an example, the actual and estimated lane changing states corresponding to test data set of driver 2 (based on trained model of driver 2) from both models are plotted in Fig. 8 and Fig. 9. The red dotted lines represent the actual driving states, while the blue lines are the estimated states. The different driving states are represented in the vertical axis, whereby 1 is LCR, 2 is LK, and 3 is LCL, while the horizontal axis represents the time length of the drive in seconds,s. The data are recorded every 0.05 s. The figures show the proximity between the actual and estimated states.
As mentioned, a generability test is performed as well in which the results are presented in Table 6 and Fig. 10. In this test, the trained model of a specific driver is tested with test data of other drivers with the aim to analyze if the performance values are close to the values obtained in Table 5. The average values (other test data) are the average  values when the test data of other drivers are used for testing the trained model of each specific driver.
Using drivers 1 and 2 as examples for the generability test, the estimated and actual lane changing states based on test   data of driver 2 (trained model of driver 1 tested) are plotted in Fig. 11 and Fig. 12.
Based on the results in Table 5, both models generate high ACC, DR, and low FAR, with the exception of FAR keep . Based on the obtained results, it can be stated that model II has a higher performance than model I in most metrics. From this observation, it can be concluded that the prefilter application on TTC variables tends to have a positive effect on the performance. The generability test results (Table 6) show that the model II also outperforms model I here except for ACC right , FAR right , and FAR keep . However, the metric values based on Table 5 are higher.

B. COMPARISONS WITH OTHER APPROACHES
Comparisons between the proposed approach with an improved HMM approach and an ANN-based state machine approach are also part of the evaluation process. The ANN-based state machine approach is developed in [15], while the HMM is based on [6]. The prefilter thresholds are optimized as well in the improved HMM, while the ANN-based state machine use biases and weights (as ANN parameters) defined by optimization to develop estimations. Model II is used for comparisons as it has a better performance than model I. All other approaches also uses the same input variables as model II.   In Table 7, the average metric values based on each driver's test data are shown. In addition, the receiver operating characteristic (ROC) curves for the three approaches based on the different lane changing behaviors are given in Fig. 13 to Fig. 15. The area under curve (AUC) values of each method based on the different behaviors are also presented in Table 8.
From the results, the HMM-based state machine approach has a better performance than the other two approaches in most metrics. On the other hand, the individual HMM approach outperforms the ANN-based approach, except for  ACC right , FAR right , ACC keep , and DR keep . An observation based on the results are the FAR keep values tend to be high in all approaches. The ANN-based state machine approach also produces a lower DR left value compared to other HMM-based approaches. Based on the ROC curves, the HMM-based state machine model has the best performance in LCR and LCL, as the model generates the highest true positive rate corresponding to a related low false alarm rate. The AUC values of the the proposed approach are also the highest in LCR and LCL (Table 8), while the AUC of the individual improved HMM approach is the highest fin LK. In general, it can be concluded that the proposed approach improves the performance of the individual improved HMM approach and the ANN-based state machine approach showing its effectiveness.

VI. SUMMARY AND CONCLUSION
Driving behavior prediction accuracy often rely on the structure of ML approaches or the input features used. In this contribution, a new approach combining a previously developed state machine model and an improved HMM for the recognition of lane changing behaviors is introduced. The state machine models the lane changing behaviors using three states, such that the model can express the transition between states or remain in the same state for estimations of the final VOLUME 10, 2022 lane changing behaviors. The modeling of the transition is realized based on specific conditions defined by the HMM estimations. Here, the HMM estimates lane changing behaviors as well. The HMM differs from a conventional HMM as a prefilter with unknown thresholds values is applied to the features used with it. Two feature sets consisting of distance as well as velocity deviation variables (model I) and TTC variables (model II) are considered for the application of this method.
Based on the results obtained, model II produces better ACC, DR, and FAR than model I. Nevertheless, both models generates acceptable results. In general, it can be concluded that using TTC variables improves the recognition performance in both experiments. A generality test is performed as part of this work, which shows that model II has a better generalization ability than model I. For further evaluations, the developed approach is compared to an individual HMM approach and an ANN-based state machine approach using model II. The results show that the HMM-based state machine model outperforms the other two methods.
RUTH DAVID is currently pursuing the Ph.D. degree with the Chair of Dynamics and Control, University of Duisburg-Essen, Germany. Her current research interests include recognition and prediction of driving behaviors based on applications of machine learning methods. She currently focuses on developing a state machine-based model as a new machine learning approach for the prediction and recognition of behaviors.
DIRK SÖFFKER (Member, IEEE) received the Ph.D. degree (Dr.-Ing.) in mechanical engineering and the Habilitation degree in automatic control/safety engineering from the University of Wuppertal, Germany, in 1995 and 2001, respectively. He has been the Head Professor of the Chair of Dynamics and Control, University of Duisburg-Essen, Germany, since 2001. His current research interests include diagnostics and prognostics, modern methods of control theory, safe human interaction with technical systems, safety and reliability control engineering of technical systems, and cognitive technical systems.