Stress Testing Method for Scenario Based Testing of Automated Driving Systems

Classical approaches and procedures for testing of automated vehicles of SAE levels 1 and 2 were based on defined scenarios with specific maneuvers, depending on the function under test. For automated driving systems (ADS) of SAE level 3+, the scenario space is infinite and calling for virtual testing and verification. However, even in simulation, the generation of safety-relevant scenarios for ADS is expensive and time-consuming. This leads to a demand for stochastic and realistic traffic simulation. Therefore, microscopic traffic flow simulation models (TFSM) are becoming a crucial part of scenario-based testing of ADS. In this paper, a co-simulation between the multi-body simulation software IPG CarMaker and the microscopic traffic flow simulation software (TFSS) PTV Vissim is used. Although the TFSS could provide realistic and stochastic behavior of the traffic participants, safety-critical scenarios (SCS) occur rarely. In order to avoid this, a novel Stress Testing Method (STM) is introduced. With this method, traffic participants are manipulated via external driver DLL interface from PTV Vissim in the vicinity of the vehicle under test in order to provoke defined critical maneuvers derived from statistical accident data on highways in Austria. These external driver models imitate human driving errors, resulting in an increase of safety-critical scenarios. As a result, the presented STM method contributes to an increase of safety-relevant scenarios for verification, testing and assessment of ADS.


I. INTRODUCTION
Testing of ADS using TFSS is an inevitable part of virtual testing procedures. The demand on test kilometers emphasized in the works of [1]- [2] and the simulationbased testing in complex virtual environments with objects, traffic, pedestrians, etc., is needed for valid and representative simulation results. The literature in virtual testing is mainly focused on scenario-based testing presented in [3]- [5], where SCS are based on a-priori knowledge coming from accident research, field operational test and expert knowledge. However, if a new system with high complexity is introduced, new failures could emerge. The presented STM inherently includes new SCS based on stochastic virtual testing. New failure patterns related to the specific ADS function are detected and could even serve as a source of knowledge for scenario-based testing. In [6], a framework for testing of ADS including a calibrated traffic flow model (TFM) for testing and generation of SCS on motorways has been presented. This simulation framework combines a vehicle simulation software IPG CarMaker with the TFSS in Vissim. This framework provides a calibrated traffic flow model (TFM) and a possible testing environment to verify the results of the STM. The TFM is behavior-based and the movement is stochastic and unpredictable with regards to the paths and decisions of traffic participants. As for this paper, we focus on motorways, SCS (e.g. collisions, near-collisions or accidents) have not occurred frequently in the used co-simulation. The reason is the lack of errors in vehicle guidance executed by TFSS, whereas those occasionally occur in manually driven vehicles. Therefore, the presented STM is used to increase the number of SCS for virtual scenario-based testing where TFM are considered in the simulation e.g. in [7]- [9]. To avoid VOLUME 4, 2016 1 arXiv:2011.06553v2 [cs.RO] 8 Dec 2020 this lack of errors, statistical accident data from Austrian motorways provided from [10] were examined for accident types that occur most frequently. These accident types were classified to longitudinal and lateral accidents and used for STM. Based on this examination, the traffic participants are manipulated to provoke SCS in the vicinity of the vehicle under test. This can be done by using the external driving model interface of Vissim described in [11] and [12]. The interface provides the possibility to implement driver models with defined driving behavior using external driver models implemented by a dynamic link library (DLL) provided in Vissim. The stress testing approach in stochastic virtual ADS testing is a development tool addressing many issues related to automated vehicle guidance, such as velocity control in ACC and AEB, LKA and automated lane change but also more complex issues such as time delays [13] and actuator failures as well as network attacks [14]. The verification of the STM method will be examined by comparing simulation results with and without using the STM.

II. SIMULATION ENVIRONMENT A. CO-SIMULATION BETWEEN CARMAKER AND VISSIM
The simulation environment is based on the co-simulation framework presented in [6] and [15]. This framework combines Matlab/SIMULINK and a co-simulation between IPG CarMaker and Vissim. In this context, IPG CarMaker is used for the implementation of the automated driving functions and sensor models for simulating machine perception as well as the visualization of simulation test runs. Additionally, it provides a complex multi-body vehicle model for a detailed representation of the ego-vehicle dynamics, as it is necessary in the non-linear region such as emergency braking or skidding. The traffic is generated by Vissim which uses a microscopic, behavioral and time step-oriented traffic flow simulation model introduced in [11] and [16]. The term "microscopically" refers to each driver-vehicle unit which can be modelled individually and therefore each vehicle has an individual driving behavior by assigning a specific set of behavior-related parameters in the driver models of each of the vehicle unit driving through the road network. For this paper, a calibrated TFM from [6] was used. The calibration is based on traffic measurements on macro-and microscopic level on the official test road for automated driving in Austria called ALP.Lab. The description and introduction to this test road is presented in [17].

B. EXTERNAL DRIVER DLL IN VISSIM
The manipulation of the surrounding vehicles for the STM is carried out by replacing internal driving behavior models with user-defined models. The DLL software framework implemented in C++ is presented in detail in [18]. In order to achieve this, defined scenarios described in the sections III-D and III-E are implemented in a DLL. During the simulation, Accidents with only one party involved  Type  Description  1  Leaving the road on the right  2  Leaving the road on the left  3  Leaving the road by a lane intersection or road exit  4  Reverse driving or turn around  5  Downfall of or in the vehicle  6 Collision with an object 7 Other accidents with only one party involved Accidents with two or more parties involved Type Description 1 Collision during overtaking 2 Lane changing with and without collision 3 Collision with a moving vehicle 4 Collision with a stationary vehicle due to traffic 5 Collision at an intersection 6 Collision due to reverse driving 7 Collision due to get into the lane 8 Other accidents with one-way traffic Vissim calls the DLL code for each affected vehicle in each simulation time-step to determine the behavior of the vehicle.

III. STRESS TESTING METHOD A. ACCIDENT DATA BASE
In order to provoke representative stress situations, a statistical database of accident data from the Austrian motorways was used to define maneuvers which are provoked in the surrounding area of the vehicle under test. Accident data has been provided by Statistics Austria and contains all accident data between 2009 and 2018. The classification of motorway accidents as defined by Statistics Austria can be found in [19]. In this paper, we focus on those classifications that are most likely to occur. Two very dominant accident classifications with their accident types are shown in Tab. 1. The total number of accidents caused by the types presented in Tab. 1 is shown in Fig. 1. These two accident classes, accidents with one party involved and accidents with two or more parties involved, are the most frequent on Austrian motorways and they comprise 94% of all motorway accidents from 2009 until 2018. From Fig. 1 it is obvious that traffic accidents where one party is involved are significantly higher and therefore, they were handled in more detail. In Fig. 2, the clusters of traffic accidents with one party involved on Austrian motorways are depicted. Based on these clusters, the four most frequent accidents types are chosen and are relevant for the STM. By abstracting these accident types in longitudinal and lateral types we define four relevant scenarios for STM, which are shown in Tab. 2.

B. TEST METHOD DESCRIPTION
The general procedure of the STM is depicted in Fig. 3. Figure 3 shows that outputs of the STM are concrete scenarios which could complement the accident database and can be used for parameter identification as in [20] and [21].     Concrete scenarios are well described in [22]. The parameter identification and methods for concrete scenario generation are not part of this paper. The traffic participants of the surrounding area of the vehicle under test are manipulated to provoke scenarios using maneuvers based on scenarios from Tab. 2. This maneuvers are called stress test events (STE).

C. STRESS TESTING FOR LONGITUDINAL SCENARIOS
In order to provoke STE in longitudinal direction, traffic vehicles T i,j as depicted in Fig. 4 will be manipulated in a defined manner.  The index i ∈ {1, 2, 3} represents the considered traffic vehicle column (TVC) and index j ∈ {1, 2, 3} the lane number of the traffic vehicle. These two index definitions are used in further equations and matrices. According to Fig. 4 the TVCs represent the locations of actual traffic vehicles in front of the EGO vehicle within a defined distance interval and 3 columns. The manipulation process for longitudinal STEs is carried out in such a way that the vehicle decelerates as described in section III-D. The distances to each d T V C k with k ∈ {1, 2, 3, max} depend on the safety interval times (SIT) t s k and current EGO vehicle speed v ego and are calculated as in (1). The dependence on the EGO vehicle speed makes VOLUME 4, 2016 Step 1 Traffic event matrix (TEM) Step 2 Event trigger conditions Step 3 Event counter Step 4 Scenario time frame Step 5 Storage of vehicle states them variable during the simulation and adjustable with the SIT t s k .
In Fig. 4 a 3-lane traffic configuration is presented, the same procedure is defined for 2-lane roads. Mandatory for the STM is the definition of distance matrices D l 2 for 2 and D l 3 for 3 lanes. The upper indices l 2 and l 3 of the matrices represent highways sections with 2 lanes and 3 lanes, respectively. The matrices are defined in (2).
They represent the distance from the EGO vehicle to each target vehicle in the TVC column and they are calculated as in (3).
In ( To implement the STM in Vissim using the described external driver DLL, the steps from Tab. 3 are defined and implemented. In the context of this work an event represents a maneuver which is provoked to bring the EGO vehicle in a stress condition. How these events are defined is described in the following steps. Step 1: Traffic Event Matrix (TEM) Depending on the number of road lanes we differentiate between two TEM which are defined in (4).
The matrix coefficients e l 2 i,j and e l 3 i,j are Boolean values and they define whether a car is in a certain TVC or not, see Fig.  4. The calculation of the matrix entries e l 2 i,j and e l 3 i,j in (5) is Requirement 1 Triggering of target vehicles T i,j can only be carried out in TVC column.

Requirement 2
The manipulation of traffic participants is triggered if the target vehicles T i,j a laying on the same lane as the ego vehicle, no matter on which TVC it occur.

Requirement 3
More then one target vehicles can only be triggered if the second or third target vehicle are in the same TVC column.

Requirement 4
The triggered longitudinal STE start first with the triggering of vehicles in the first T V C 1 column, then the second T V C 2 and then third T V C 3 column. After reaching a certain number of events in one column, these events are not triggered anymore. done by checking if the entries of distances matrices D l2 and D l3 in (2) are in the range of distances An example how of a random traffic event in Fig. 5 shows how the TEM is calculated. In the simulation the TEM is calculated continuously until a trigger condition is executed.
Step 2: Event Trigger Conditions As we consider longitudinal stress maneuvers in this chapter, the events which are triggered are braking maneuvers (BM). These BM are divided in two braking categories which could be triggered for the STM. The detailed definition of the BM is described in section III-D. Based on the TEM defined in (4) the combination matrices are defined in (6).  The matrices C l2 q and C l3 q represent relevant traffic conditions which should be tested for two and three traffic lanes, respectively. The index q represents the number of relevant braking combinations. For the triggering of braking maneuvers, the combination matrices in Tab. 5 and Tab. 6 are relevant for the STM. Entry 1 means the car was on this position and an event has occurred, so the car should start braking. Entry X could be 0 or 1 but the vehicles on this position will not brake. According to these tables, the combination matrices are examined by satisfying the requirements form Tab. 4. The braking maneuvers will be triggered only if in at least one TVC column a traffic vehicle occur and only in a situation when the traffic vehicle is located in the same lane as the ego vehicle. By comparing the event matrices E l2 T or E l3 T with the combination matrices C l2 q and C l3 q an event is triggered if the matrix entries match. The event shown in Fig. 5 would yield to the matrix entries shown in (7).
If such a event occur, the trigger time t trigger is saved and an event counters n l2 ct and n l3 ct are incremented.
Step 3: Event Counter Each event which occurs is counted by incrementing the trigger counter n l2 ct or n l3 ct . To make sure that all events happen at least a certain number of times a maximum number of occurrences n max ct during a simulation run is defined and is used as parametrization parameter of the STM. That means, if a certain combination from the matrices C l2 q or C l3 q occur n max ct -th times they will not be triggered any more during a simulation run.
Step 4: Scenario Time Frame Each detected scenario by the STM is saved within a certain time frame. This time frame is defined by the trigger time t trigger , the upper frame limit t upper and lower frame limit t lower . The upper-and lower-time frame limits can be adjusted and together with the trigger time t trigger they represent the scenario time frame t scene shown in (8).
Step 5: Storage of Vehicle States In case that a scenario is detected all data listed in Tab. 7 will be saved for further analysis. The stored data will contain all vehicles in the scenario time frame t scene . This includes only the relevant vehicles depicted in Fig. 4.

D. ACCELERATION AND DECELERATION BEHAVIOUR
In Tab. 2, longitudinal scenarios with a stationary vehicle in front are relevant scenarios for the STM. These types of accidents can be caused by full braking maneuvers or sudden speed reductions on the motorway. The type of accidents where a vehicle is approaching an existing crash will not be considered since this type of accidents is exhausted in various research and industry works in the field of active safety systems. For more details about active safety systems, see [23]- [27]. Considering that, braking behavior of vehicles during the simulation has a significant impact on the validity of the results of the STM. As described in section II-B, we use the external driver DLL in Vissim to redesign the deceleration characteristics, which will be more effective to improve the realistic simulation of the vehicle. Majority of studies in the past have proposed deceleration models, Akçelik and Biggs suggested non-uniform deceleration rate VOLUME 4, 2016 to describe a polynomial behavior between acceleration and speed [28]. However, since previous models were limited to the study of cars and trucks only, dual regime models depict the deceleration behavior of various vehicle types that were proposed in [29] and [30]. Some studies demonstrate that deceleration profile can be presented using a polynomial fit which shows that the maximum deceleration typically occurs during the braking stabilization phase on ESP-equipped cars [32], therefore, we can simplify the deceleration profile into a polynomial model. As described above, previous studies have established that the basic deceleration profiles can be fitted in a polynomial model, thus the deceleration for different traffic scenarios needs to be discussed as well.

Deceleration gradient
The relationships between deceleration, jerk value and speed are specified by the ISO 22179 in [33]. Those specifications are considered for the calculation of the deceleration gradients for the STM. Current research is more focused on classifying the deceleration models and refining the deceleration characteristics for different functions and different scenarios, e.g. ACCequipped cars. The deceleration and acceleration gradient rate are strictly limited to meet the requirements of comfort function as defined in [33]. Unlike the ACC functional features, the driver braking reacts differently to complex traffic conditions and has more variable deceleration characteristics. With the electrification of vehicles increasing and the traffic environment becoming more complex, different deceleration models need to be classified so that they can be more relevant to the actual traffic conditions in the simulation. In general, the average deceleration will be based on experimental data, taking into account different driving styles and scenarios. A general conclusion can be drawn from the experimental statistics under normal driving conditions, the deceleration rate will not exceed -3.5 m/s 2 , refer to [34]. To refine the scenario, traffic environment and acceleration data for different drivers, the deceleration and braking process is represented using a parabolic model from [35] combined with a maximum deceleration table, where the deceleration parameters can be based on different factors (e.g. gender, traffic flow, traffic conditions, age, etc.). This provides a reference to calibrate deceleration rate, braking distance, and duration time. Deceleration behavior model is divided into two categories. Tab. 8 shows the driver braking model which is used to simulate the braking characteristics of a real driver, and Tab. 9 shows the ACC braking model which is used to simulate the deceleration profile with ACC driving function activated.

1) Driver Braking Behavior
Past studies about driver behavior proposed that vehicle needs more time and distance to decelerate at high speed. In this sense, the vehicle deceleration rate at high speed is much higher than at low speed. In order to cooperate with the simulation of vehicle dynamic characteristics at high speed, the deceleration rate is determined according to the speed range and target speed as an input condition which is shown in Tab   The driver braking model using external driver DLL interface is integrated into Vissim for testing. Fig. 6 shows the acceleration rate and vehicle velocity based on the polynomial model for a vehicle decelerating from the initial velocity 71.03 km/h to final velocity v f inal = 28.67 km/h. The deceleration time was set to 12 s and the maximum deceleration at -1.71 m/s 2 .The results of Vissim simulation allow the observation of a slight oscillation around a constant velocity profile, which also corresponds to the real performance of the velocity behavior. The profile of driver braking model will drop rapidly to maximum deceleration in the beginning phase and then gradually return to 0 with the velocity decreasing.

2) ACC Braking Behavior
Since ACC is a comfortable longitudinal control function, the limitations of execution range in [33] are used for the development and implementation of an ACC braking model. The deceleration gradient rate is limited in the standard, as shown in Fig. 7. The average rate of deceleration gradient cannot exceed 2.5 m/s 3 when the speed is greater than 20 m/s and 5 m/s 3 when it is less than 5 m/s. The deceleration gradient is determined based on the initial vehicle speed. The entire deceleration process is expressed in the form of a parabolic as shown in Tab. 11. The implemented ACC braking model in Vissim as depicted in Fig. 8 shows that the profile is determined based on the parabolic model. Jerk value is set to -1.5 m/s 3 and maximum deceleration to -3 m/s 2 at the time of simulation. The initial velocity is reduced from 70.97 km/h to 42.25 km/h considering the maximum deceleration gradient rate. In comparison with driver braking model, the deceleration gradient becomes a key factor to limit ACC deceleration profile. Therefore, the whole braking process is derived by a parabolic model.

E. STRESS TESTING OF LATERAL SCENARIOS
The second type of scenario considered for the STM are the lateral movements of traffic participants. As seen in Tab. 2, accidents caused by unappropriated lateral behavior of traffic participants are second dominant in the data base of accidents. These accidents occur while a traffic participant initiates an aggressive cut-in maneuver from the left or right sight of a certain vehicle on road. In our case, the ego vehicle drives through the traffic and in case a traffic participant is overtaking or driving past the ego vehicle on the left or right lane. The cut-in maneuvers which are handled and manipulated by the STM are depicted in Fig. 9 and 10. The times t LCE init,i with i ∈ {1, 2, 3} from Fig. 9 and 10 represent the initialization times of an aggressive lane change events (LCE). An LCE starts with initialization time t LCE init,i and ends with the maneuver time t m . During the simulation, each lane change event is executed one after the other with interval time t int i,j between two lane change events. The minimum interval t int 1,2 is set to 5 minutes. After these 5 minutes the next LCE is executed only in case a scenario depicted in Fig. 9 and Fig.  10 occur. After the last initialization time t LCE init,3 the events occur again from the beginning t LCE init,1 . The description of this is shown in Fig. 11. The initialization times t LCE init,i and the event times t int 1,2 are variables and can be adjusted within the Vissim interface. The lane change behavior and maneuver itself is described in the next subsection.

F. LANE CHANGE BEHAVIOR
The implemented lane change behavior in Vissim is based on the lane change algorithm described in [39] and [40].  Tab. 12. To match more realistic lateral scenarios, the acceleration profile is calibrated with experimental data used in [40]. The entire process of lane change has an acceleration at the beginning phase and a gradual decrease in speed after the maneuver is completed. As a result, a sinusoidal function is involved to present lane change acceleration profile and shown in Tab. 12.
Lane change trajectory equation  The result calculated from lateral and longitudinal trajectory equations will be used as a reference to verify that Vissim output can follow the input commands. As an example, for a possible lane change scenario, a lane change maneuver is built with the longitudinal vehicle velocity v m = 85 km/h. The total lane change maneuver time is set to t m = 6 s, which corresponds to an average maneuver time according to [41]. The lateral displacement y lat (t) in the Vissim road model from [6] is h=3.5 m. This represents the distance from one center line to another. Regarding the longitudinal behavior, the maximum acceleration value is set to a max =1.2m/s 2 . The trajectory equations for this example can be obtained in (9), (10) and (11).
y lat (t) = −0.0067t 5 + 0.0840t 4 − 0.2800t 3 (9) y long (t) = 85t (10) The comparison between the calculated values y lat to the values y V lat set by Vissim using the DLL is shown in Fig. 12. A minor deviation from the desired values can be observed.
The deviations are caused by Vissim internal cycle times (min. program cycle 0.5s) and settling times which cannot be manipulated. For the STM only the cut-in in front of the ego vehicle is important for the testing process. After a target vehicle finishes the lane change, the driver behavior is not controlled by the DLL. The traffic vehicles continue to drive on the road using internal Vissim driver models.

IV. RESULTS
To demonstrate the effectiveness of the STM, the cosimulation framework developed in [6] is used. The ego vehicle is an internal IPG Driver which is equipped with an ACC and automatic lane change algorithm. For the increase of the number of scenarios relevant for testing of ADS, the Vissim traffic is featured with the STM using an external driver model DLL interface. For this purpose, a Driver Model framework was developed and presented in [18] to manipulate traffic vehicles in Vissim in order to provoke critical scenarios. Therefore, two simulation runs with 5000 kilometers are carried out. The first simulation run is carried out without the manipulation procedure and the second simulation run contains the manipulation according the presented STM. Critical scenarios which should be detected and evaluated, collisions and critical scenarios as defined in [42] are taken to prove the efficiency of the STM. Based on the data from [42] time-to-brake (TTB) and the requested acceleration are used to assess the detected scenarios. In [42] three criticality levels are defined, non-critical, eventually critical and very critical. One possible and exemplary STM parameter configuration chosen by the engineer judgment for this research is defined in Tab. 13. Tab. 14 shows that the number of detected collisions increased from 59 to 625 on 5000 km. According to the metrics defined in [42] very critical and eventually critical scenarios increase significantly compared to the cosimulation without the STM. Other parameter configurations are allowed and will be used for parameter studies in future research.

V. CONCLUSIONS
In this paper we presented a stress testing method to increase and generate safety critical scenarios in a complex cosimulation between IPG CarMaker and PTV Vissim. Based on statistical accident data for motorways in Austria, two dominant accident types of scenarios are extracted and used for the presented method. Using these accident types, defined maneuvers are provoked by traffic participants in the surrounding area of the vehicle under test. The developed stress testing method showed a significant increase of detected scenarios in the co-simulation environment and will be used for further research and development. In addition to manipulating traffic vehicles on the motorway, a huge potential using the presented method lies in the use for urban areas where the environment is far more complex than on motorways. HEXUAN LI received the B.S. and M.S degree from University of Clermont Auvergne, France, 2016, both in mechatronics engineering. From 2016 to 2020, he was employed at PSA and Audi China successively, mainly responsible for the highly automated driving system integration and hardware-in-the-loop simulation. Since 2020, he has been working with the Institute of Automotive Engineering, University of Technology Graz, dealing with Driver Assistance Systems, sensor modelling and co-simulation framework based on a full vehicle test bench.
ARNO EICHBERGER received the degree in mechanical engineering from the University of Technology Graz, in 1995, and the Ph.D. degree (Hons.) in technical sciences, in 1998. From 1998 to 2007, he was employed at MAGNA STEYR Fahrzeugtechnik AG&Co and dealt with different aspects of active and passive safety. Since2007, he has been working with the Institute of Automotive Engineering, University of Technology Graz, dealing with Driver Assistance Systems,Vehicle Dynamics and Suspensions. Since 2012, he has been an Associate Professor holding a venia docendi of automotive engineering.
CHRISTOPH WELLERSHAUS received the BSc degree in mechanical engineering and business economics from Technical University of Graz, Austria, in 2019. He is currently pursuing his MSc degree in mechanical engineering and business economics at Technical University of Graz, Austria. He started to work as a scientific student assistant in the research field of driver assistance and automated driving at the Institute of Automotive Engineering at Technical University of Graz in 2018. From 2016 to 2018 he participated in the Formula Student -student association "TU Graz Racing Team", where he held the role of the team leader from 2017 to 2018.
ALEKSA PANDUREVIC received the BSc degree in computer science from the Graz University of Technology, Graz, Austria, in 2020. Currently, he is studying for a MSc computer science degree at the Graz University of Technology and Technical University of Munich. In 2019, he joined the Institute of Automotive Engineering at Graz University of Technology as a Student assistant on scientific projects where he contributed to different projects in field of automated driving and driver assistance systems. Since 2020, he also works as a software development intern at Infineon Technologies Austria AG, Graz. His current topics of interest are machine learning, data science and information security.
BRANKO ROGIC recived his bachelor and master degree in mechanical engineering from the Technical University of Graz. His final degree project dealt with the simulation of hybrid commercial vehicle drivetrains and was carried out on the Institute of Automotive Engineering. After the graduation he worked as a development engineer at the automotive company ZF (in 2013 and 2014). 2015 he returned to the Institute of Automotive Engineering to work on an industrial research project, dealing with the integration of ADAS functions and in cooperation with Magna Steyr. Since 2018 he is an employee by Magna Steyr at the ADAS advanced development department. VOLUME 4, 2016