Driver Response to Take-Over Requests in Real Traffic

Existing research on control-transitions from automated driving (AD) to manual driving mainly stems from studies in virtual settings. There is a need for studies conducted in real settings to better understand the impacts of increasing vehicle automation on traffic safety. This study aims specifically to understand how drivers respond to take-over requests (TORs) in real traffic by investigating the associations between 1) where drivers look when receiving the TOR, 2) repeated exposure to TORs, and 3) the drivers’ response process. In total, thirty participants were exposed to four TORs after about 5–6 min of driving with AD on public roads. While in AD, participants could choose to engage in nondriving-related tasks (NDRTs). When they received the TOR, for 38% of TORs, participants were already looking on path. For those TORs where drivers looked off path at the time of the TOR, the off-path glance was most commonly towards an NDRT item. Then, for 72% of TORs (independent on gaze direction), drivers started their response process to the TOR by looking towards the instrument cluster before placing their hands on the steering wheel and their foot on the accelerator pedal, and deactivating automation. Both timing and order of these actions varied among participants, but all participants deactivated AD within 10 s from the TOR. The drivers’ gaze direction at the TOR had a stronger association with the response process than the repeated exposure to TORs did. Drivers can respond to TORs in real traffic. However, the response should be considered as a sequence of actions that requires a certain amount of time.

an extent that drivers do not need to supervise the vehicle's performance and may, therefore, engage in other activities (e.g., using a mobile phone; [1], [2]).Consequently, drivers may be out of the loop as they are allowed to not be in physical control of the vehicle and not monitor the driving situation [3].A current concern is whether drivers who are out of the loop can respond appropriately to take-over requests (TORs) and resume manual control safely as assumed by the system specifications [1].It is important to understand how to design TOR procedures that enable safe transitions of control from AD to manual driving to achieve safe vehicle automation.The design of TOR procedures are guided by regulations on automated lane keeping system (ALKS) (United Nations Economic Commission for Europe: [4]).Current regulations on ALKS state that drivers should be given at least 10 s to deactivate AD before a minimum risk maneuver needs to start [4].A minimum risk maneuver is defined as "a procedure aimed at minimizing risks in traffic, which is automatically performed by the system after a transition demand without driver response or in the case of a severe ALKS or vehicle failure," [4].).

A. How do Drivers Respond to TORs?
The ability of drivers to respond to TORs has been extensively studied in driving simulators of various fidelities (fix-based, motion-based; [5]).In these studies, the most common way of assessing how well drivers can respond to TORs is to measure the take-over time: the time it takes the driver to react to the TOR and deactivate AD by braking, steering, or pressing buttons [5], [6], [7].However, to obtain a complete understanding of drivers' response to TORs, it is important to investigate other parts of the response process, rather than just measuring the take-over time.While take-over time indicates that a driver succeeded with the deactivation of AD, assessing additional responses (visual and motor) can provide more information about a driver's readiness to perform safe manual driving such as whether a driver has looked on road or moved their feet back to the pedals.With this in mind, a subset of driving simulator studies have also included response times for: 1) redirecting the gaze away from an NDRT item (e.g., [8]); 2) the first on-road glance [8], [9], [10]; 3) placing hands on the steering wheel [8], [9], [10], [11]; and 4) glancing towards mirrors [8], [12].Ideally, when a driver has resumed manual control, she should be sufficiently in the loop: "in physical control of the vehicle and monitoring the driving situation" [3], to be able to safely respond to the changing traffic situation.Thus, understanding how to design TOR procedures This work is licensed under a Creative Commons Attribution 4.0 License.For more information, see https://creativecommons.org/licenses/by/4.0/Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
that enable safe transitions of control from AD to manual driving requires a combined analysis of drivers' physical and visual behavior.

B. Influence of Engagement in NDRTs and Repeated Exposure to TORs on drivers' Responses to TORs
Engagement in an NDRT during AD can involve a handheld or a mounted device.Previous driving-simulator research indicates that take-over times are longer when drivers are engaged with handheld items than with mounted items or no NDRT [5], [6], [10].Furthermore, it seems that this increase stems from increases in both visual response times (e.g., redirecting gaze away from task; [12]) and physical response times (placing hands on wheel; [11], [12]).One clear reason for the increase in take-over time when an item is handheld is that drivers need to put away the item, which may also require visual scanning to find a suitable place, before deactivating AD.Less is known about the influence of NDRT engagement on the time to move the foot back to the pedals (brake or accelerator), especially in real traffic.As holding an item and moving the foot are not competing tasks (i.e., they can be performed concurrently), there may not actually be a delay in moving the foot to pedals when the driver is engaged in NDRTs.
Another factor that has been found to influence take-over times is practice of the TOR procedure.In fact, Zhang et al. [6] found that take-over time decreased on average 1.1 s between the first and second take-over events, while Gold et al. [13] suggested the relationship between repeated exposure to TORs and take-over times to be logarithmic (i.e., the take-over time decreases less with each exposure).Furthermore, one study found that repeated exposure shortened both the time required to look on road (first repetition compared to fourth) and the time to put hands on wheel (first repetition compared to third and fourth) [14].

C. Need to Validate Previous Findings in Real Traffic
As most previous research on drivers' response to TORs has been conducted in virtual settings, there is a need to validate these findings in real traffic with a real vehicle.Furthermore, driving-simulator studies tend to focus on critical scenarios: a TOR is typically issued at 7 s time-to-collision at high speeds (120 km/h or higher; [5], [15]).Given the bias in the literature towards highly critical scenarios in driving simulators we may underestimate drivers' ability to respond to TORs in naturalistic settings (i.e., we may believe it is worse than it is).In fact, in a recent paper by de Winter et al. [15], the realism of these critical scenarios was questioned: Will these very critical scenarios preceded by a TOR even occur in future AD?In fact, it seems reasonable to believe that future AD that meet certain safety requirements would be able to issue TORs well in advance of required control transitions.For example, an AD using GPS will know when the vehicle is about to exit the operational design domain (ODD), such as leaving a highway, more than 7 s before.Therefore, there is a need to extend current findings (from the virtual environment involving critical scenarios) with studies conducted in real traffic in which drivers are given more than 7 s to deactivate AD.To the knowledge of the authors, only two studies have investigated drivers' responses to TORs during AD (hands-free driving allowed) in real traffic and under noncritical conditions (see, [16], [17]).Both these studies allowed participants to engage in NDRTs (as a manipulated variable in [16] and as voluntary engagement in [17]).In both studies, drivers managed to transition back to manual driving in response to TORs.However, Rydström et al. [17] reported take-over times up to 25 s, whereas Naujoks et al. [16] only report take-over times up to 10 s.The likely explanation behind the longer take-over times in [17] is that the participants had no prior experience of the TOR and deactivation strategy prior to the test, while the participants in [16] had practiced beforehand.

D. Aim and Research Questions
The following study aims to advance the current understanding of drivers' response to TORs in real traffic with a real vehicle, using a prototype TOR procedure that requires a steering button press to deactivate automation.Specifically, it explores how the driver's response process is influenced by repeated exposure to TORs and where drivers look when the TOR is issued.This study also breaks down the response process into a series of actions and measures the times taken to execute each action in response to the TOR.These times can serve as reference values in future work and help validate results from virtual environments.To achieve these aims, the following research questions were specified: In real traffic, 1) How do drivers respond to TORs? 2) Is there any association between: a) the repeated exposure to TORs; or b) the gaze direction when the TOR is issued (at an NDRT item versus on path) and the driver response to the TOR? 3) When drivers who are looking (at an NDRT item versus on path) receive a TOR, what are the expected response times for the necessary actions?II.METHOD

A. Participants
In total, 30 participants (10 females and 20 males), all employed at Volvo Cars in Gothenburg, participated in the study.They had no direct involvement in the development of AD, but many of them had some familiarity with driving assistance systems: 83% used an adaptive cruise control and 50% used a lane keeping aid on a regular basis.The mean age was 39.1 (SD = 10.5) years and the mean driven mileage during the last year was 16583 km (SD = 13208).This study was approved by the national ethical review board in Gothenburg, Dnr: 2019-01827.All participants signed a consent form prior to participation.

B. Testing Environment and Equipment
The study was conducted on a public ring road with real traffic in Gothenburg, Sweden.One lap was approximately 30 km long, and the posted speed was 70 or 80 km/h.The road was mainly a divided highway with 2-3 lanes in each direction.A map showing the ring road and a picture showing the traffic environment can be found in Pipkorn et al. [18,Fig. 1].This study took place during the daytime on weekdays, during off-peak hours, to ensure free-flowing traffic.A Wizard-of-Oz (WoZ; [19]) test vehicle was used in the study to simulate an AD feature.The vehicle included a set of pedals and a steering wheel in the mid position of the rear seat, visually obstructed from the test participants.Thus, automation was simulated by a wizard driver who drove the vehicle from the rear seat.The wizard's head and shoulders were visible from the front seat.The wizard's presence was explained to the participants as a safety-measure; he would supervise the automation and only intervene if needed.The test vehicle was equipped with cameras that recorded video data (10 Hz), capturing the driver's face, feet, and upper body along with the forward roadway.
1) The AD Feature: The ODD of the simulated AD included safe driving in lane (complete operational control and event detection and response) under good weather, lighting, and road conditions.This specific AD was not able to handle traffic lights or lane changes.When AD was available for activation, the system notified the driver with an audio tone and a message in the instrument cluster reading, "Autopilot available".The driver could then activate AD by pressing two buttons on the steering wheel for 0.6 s (after the button press, the vehicle control shifted to the wizard driver).The buttons were located so that the thumbs could reach them when the hands were on the steering wheel.The participants received feedback when the AD was activated, with a voice saying, "Autopilot active" and an updated view in the instrument cluster (see Fig. 1, bottom).
The TOR consisted of an audio tone (distinguishable from the tone used when AD was available for activation) and a message in the instrument cluster reading, "Autopilot ending" (see Fig. 2, top).From the time the TOR was issued, the participants had 6 s to deactivate AD (the time was displayed in the instrument cluster with a shrinking red bar; see Fig. 2, top).When a participant needed longer than 6 s, the wizard would simply continue driving until the participant deactivated AD.Deactivation was performed by pressing the same two buttons previously used to activate the system for 0.6 s.The remaining time until deactivation was visualized in the instrument cluster In both views, the information "Hold steering wheel buttons to take control" is displayed.
with two turquoise bars (see Fig. 2, bottom) approaching each other and meeting when the deactivation was completed.When AD was deactivated, the instrument cluster view changed to the manual driving mode view (see Fig. 1, top) and a voice said, "Drive the car".

C. Study Procedure
Prior to the test drive, all participants were asked to read an information sheet about the driver's responsibilities during manual driving and AD.The drivers were requested to obey traffic rules (e.g., observe speed limits) when in manual mode, and drive as they normally would without using any driver assistance systems.The participants were informed about the AD's ODD.Specifically, they were told that when AD was activated, the vehicle was fully responsible for all aspects of the driving task (including lane keeping), except when approaching a traffic light and when lane changes were required to stay on the selected route.They were informed about the need to deactivate AD if they received a TOR, as well as the required button press to deactivate the system.They were also informed that the vehicle would keep on driving until they deactivated the system (i.e., the AD did not perform a minimum risk maneuver, but continued driving in the lane until the drivers responded to the TOR).Before the actual test started, they practiced activating and deactivating AD several times, both at stand-still and during a short drive.The participants had the opportunity to bring items (e.g., magazine, notebook, and phone) of their choice to use when AD was active.

D. Test Drive
A test leader was present in the test vehicle with the participant and the wizard driver.The test leader gave the participants directions (where to drive, when to turn AD on), but no other conversation took place.The participants drove the two laps with Each lap started with manual driving, followed by AD and a practice TOR.Each of the four test AD drive segments ended with a TOR (TORs 1-4).a combination of manual driving and AD.The total duration was about 60 min.During the test drive, the participants experienced a total of four test segments with AD of about 5-6 mins each, which all ended with a TOR.
In addition, at the beginning of each lap, the participants had a 1-min practice session in AD with a TOR, to familiarize themselves with the system and the TOR.Fig. 3 illustrates the segments of TORs and manual driving.The locations of the TORs were the same across participants and laps.TORs 1 and 3 (same location) were issued prior to the need to exit one highway and enter another.TORs 2 and 4 were issued before a lane change was required to stay on the selected route when the highway diverged.

E. Data Processing and Analysis
A total of 120 video segments (four per test drive), which included the moment of each TOR and the 10 s that followed, were selected.The participants' gaze direction at the time of the TOR as well as their response process to the TOR were manually coded.The response process was broken down into six actions: first glance to instrument cluster, first glance on path, hands on wheel, foot on brake pedal, foot on accelerator pedal, and automation deactivation.The manual coding was performed by an analyst who screened the video segments using the video views shown in Fig. 4. The resulting annotations were reviewed either by another independent analyst or by the first author to ensure high quality.
The gaze direction at the time of the TOR was coded as On path if towards the forward roadway, NDRT if an ongoing glance towards a handheld NDRT item (e.g., mobile phone or water bottle) or towards the in-vehicle mounted tablet, or Other if an ongoing glance elsewhere, such as toward mirrors, instrument cluster, or any other exterior/interior peripheral areas.Any glance that could not be determined because the participants' eyes were not visible was coded as Unknown.
Out of the initial 120 events (30 drivers and 4 repetitions), one event was excluded due to issues with the TOR, and four events were excluded due to missing glance data (eyes not visible on video).The resulting dataset used in the analysis included 115 events.The response process for each driver and each exposure was visualized for 0-10 s after the TOR.In addition, descriptive statistics were used to show the frequency of certain responses (e.g., braking) and the order of the actions within the response process.A safety rating of the overall transition, including the response to TOR and the driving performance after automation deactivation, was performed using the TOC rating scheme [20].As 97.5% of transitions received the highest safety score (and the remaining 2.5% the second highest), no further analyses of the driving performance were conducted after automation deactivation.

1) Statistical Modeling:
To assess the associations between the drivers' response process and a) repeated exposure to TORs and b) gaze direction at the time of the TOR, response times to the TOR were computed by anchoring the following four actions at the TOR: first glance to instrument cluster, hands on wheel, foot on accelerator pedal, and automation deactivation.Note that two of the actions comprising the response process were not included: the time taken to move the foot to the brake pedal was not considered, since few drivers put their foot on the brake pedal within 10 s of the TOR.The first glance on path was not included either since the response times would consist of zeros when the gaze direction at the time of the TOR was already on path.
Response times to the TOR were modeled using Bayesian generalized linear models with intercepts that varied by participant, a modeling strategy that can demonstrate both individual variation and model convergence.In general, the modeled dependent variable was the response time, and the independent variables were: a) gaze direction at the TOR (On path or NDRT); and b) repeated exposure to TORs 1-4.Further criteria for an event to be included in the modeling were: a) the response time could not be zero (i.e., the driver could not already have performed the action); b) the event needed to include a gaze direction that was on path or towards a NDRT item at the time of the TOR; and c) glance data existed for the event.There were 91 events that met these criteria for first glance to instrument cluster, hands on wheel and automation deactivation, and 80 that met the criteria for foot on accelerator.
The models were specified in accordance with McElreath [21], as follows: where RT i is the response time (i.e., the outcome variable), which is modeled as lognormal with a mean μ i and variance σ. μ i is the model mean, which depends on the model-intercept (μ α + sd α α participant[i] ) representing the average response time for a specific participant when gaze direction at TOR = On path and repeated exposure = 1.We used a noncentered parametrization to improve the model sampling [21].This means that the model-intercept consists of α participant[i] , which is sampled from a standard normal distribution and then scaled by the subjectspecific standard deviation sd α , and added to the population mean μ α .μ i also depends on the slopes β REP2 , β REP3 , and β REP4 that represent the deviations from the model-intercept for repetition = 2, 3, or 4 and the slope β NDRT that represents the deviation from the model-intercept when looking at a NDRT item at the TOR.X REP2 , X REP3, X REP4 , and X NDRT are dummy-coded variables including a 1 for rows where the exposure equals 2 for X REP2 , 3 for X REP3 , 4 for X REP4 , and the TOR gaze direction equals NDRT for X NDRT and coded as 0 otherwise.
Priors were set on each parameter using prior predictive checks, which ensured the ability of the priors to generate plausible response times.The intercept was centered at the sample mean for the corresponding driver response time, in line with the work of Westfall and Yarkoni [22].The analyses were performed using Python ver.3.7.6 and the probabilistic programming library PyMC3 ver.3.9.3[23].The Markov Chain Monte Carlo algorithm No-U-Turn Sampler was used to fit the models [24].All models were fitted with four Markov Chain Monte Carlo chains.In total, 3000 samples were drawn from the posterior distribution for each chain (after 2000 samples had been used for tuning the sampler and then discarded).The model convergence was verified through visual inspection of the generated trace plots.Finally, each model's goodness-of-fit was assessed by comparing the posterior predictive distribution against empirical data.The posterior predictive distribution is the distribution of new values predicted by the model, given the old data.Detailed information about the modeling, including the model output and model validation, can be found in this study's supplementary material.
The models were summarized with mean and 95% highest posterior density (HPD) intervals for the parameters β REP2 , Fig. 5. Subset of the drivers' response processes to each of the four TORs.The complete figure including all participants' response processes can be found in this study's supplementary materials.
β REP3 , β REP4 , and β NDRT.These parameters represent deviations from the No NDRT/1st exposure condition.A larger deviation (positive or negative) means that the condition has a greater effect on the response time than a smaller deviation: a negative deviation means a decrease in response time and a positive deviation means an increase in response time.A situation with a 95% HPD for β that includes zero suggests a weaker association between parameters than one that excludes zero [19].However, it is important to remember that the output of the Bayesian analysis is always the complete posterior distribution, which can be used by readers and system designers to apply their own specific thresholds in making decisions.Finally, the models were used to generate posterior predictive distributions for the fourth exposure and for the gaze direction at the time of the TOR when it equals either NDRT or On path, for both 50th and 95th percentile drivers.The 95th percentile driver was included because in TOR design it is important to consider not only mean values but also more extreme values, as these typically correspond to more safety-critical behaviors [25].The distributions for 50th and 95th percentile response times for gaze directions NDRT or On path were summarized with the median and the 95% HPD.Given the data and the model, the new predicted data represent the best guesses for future response times, to be used as reference values for drivers' responses to TORs in future AD vehicles.

III. RESULTS
Participants' responses to the TOR varied both in timing and in the order of the actions (see examples of participants' response processes in Fig. 5).At the time of the issued TORs, 41% (47/115) of participants were gazing towards an NDRT item (either a hand-held item or the mounted tablet); see examples of these responses with a filled black half-circle in Fig. 5.When the remaining TORs were issued, the participants were either gazing on path (38% or 44/115) or towards other off-path areas including exterior/interior peripheral areas, vehicle mirrors and the instrument cluster (21% or 24/115).As noted, only in a few cases (7% or 8/115) did participants put their foot on the brake Fig. 6.Frequency of response-process sequences divided by Gaze direction at TOR (On path or Off path: rows) and First action (Looking on path, Looking to instrument cluster, or Hands on wheel: columns).For sequences in which a participant already performed one of the actions at the time of the TOR, the action is in parentheses.The three most frequent sequences are marked with bold numbers 1 to 3. Sequences where drivers deactivate automation before looking to the forward road are marked with a black box.A "/" indicates that actions occurred at the same time.pedal within 10 s of the TOR (see response for participant 4 as an example in Fig. 5).

A. Response Process Sequences
There was no single TOR response-process sequence (i.e., order of actions) that was common for all participants.Instead, participants responded to the TOR in a variety of ways.When actions that had already occurred at the time of the TOR were included (e.g., foot already on the accelerator pedal or gaze already on path when the TOR was issued), we observed 24 different types of sequences for the 115 issued TORs (see Fig. 6 for a summary).The top row in Fig. 6 presents sequences in which a participant already had their gaze directed on path at the time of the TOR: the star marking the first glance on path is in parentheses.In all these sequences, the participants responded to the TOR by directing their gaze towards the instrument cluster.Note that in seven out of the 44 sequences in that row, the participants also already had their foot on the accelerator pedal when the TOR was issued (yellow star and purple square both in parentheses).The bottom row in Fig. 6 presents sequences in which participants were looking off path at the time of the TOR.Participants had their foot on the accelerator pedal at the time of the TOR in five sequences (two purple squares in parentheses in column First action: Looking on path and three in column First action: Looking to instrument cluster).Participants were already looking at the instrument cluster at the time of the TOR in four sequences (three blue left arrows in parentheses in column First action: Looking on path and one in column First action: Hands on wheel).Participants were both looking at the instrument cluster and had their foot on the accelerator pedal when the TOR was issued in two sequences (one blue left arrow and one purple square in parentheses in column First action: Hands on wheel).Importantly, when a participant was looking off path at the time of the TOR (n = 71), the first response to the TOR was most often to look towards the instrument cluster before looking on path (55% or 39/71), rather than the other way around (37% or 26/71).
It is worth mentioning that most actions in the response process can be performed independently and concurrently, except for automation deactivation, which cannot occur until drivers have placed their hands on the steering wheel.That is, in Fig. 6, the red right arrow always occurs to the right of the green circle.Furthermore, a sequence starting with hands on the wheel was rare and only occurred in three cases, when participants were already looking towards the instrument cluster when the TOR was issued.In fact, if a participant was not looking towards the instrument cluster at the time of the TOR, putting hands on wheel always occurred after the participants looked at the instrument cluster.As seen in Fig. 6, the green circle always occurs to the right of the blue left arrow in columns First action: Looking on path and First action: Looking to instrument cluster.Finally, participants always responded to the TOR (i.e., not considering sequences with actions already occurring at the TOR) by either looking on path, looking towards the instrument cluster or placing the hands on the steering wheel before performing the other actions.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

1) Most Common Sequences:
The most common response process sequence (present in 30% (35/115) of events) was: 1) looking on path; 2) looking towards the instrument cluster; 3) putting hands on wheel; 4) deactivating automation; and 5) putting the foot on the accelerator pedal (see sequences marked with superscript 1 Fig. 6).Note that these 30% include sequences when a participant was already looking on path at the time of the TOR (see top left corner in Fig. 6) and when a participant was looking off path (see bottom left corner in Fig. 6) but responded to the TOR by first looking on path.The second-most common sequence (present in 21% (24/115) of the events) was the same as the first except the automation deactivation and foot on accelerator pedal changed places (see sequences marked with superscript 2 in Fig. 6).The third-most common sequence (present in 12% (14/115) of events) was: 1) looking toward the instrument cluster; 2) putting hands on wheel; 3) looking on path; 4) deactivating automation; and 5) putting the foot on the accelerator pedal (see sequence marked with superscript 3 in Fig. 6).In sum, 63% of the events are represented by five of the 24 sequences.
2) Rarely Occurring Sequences: The remaining 37% of events are represented by 19 different sequences.Examples are sequences in which participants perform multiple actions at the same time (represented by a "/" in Fig. 6) and sequences in which participants had already performed one or two actions by the time of the TOR.Importantly, in 5% (6/115) of sequences, participants deactivated automation before having looked on path (these events are marked with a black box in Fig. 6).In all these events, the participants were looking towards an NDRT item at the time of the TOR.Thus, in some cases participants had not looked on path just before the TOR until after they had deactivated automation.

B. Association Between Repeated Exposure to AD or the Gaze Direction at the Time of TOR and the Response Process
In 91 of the initial 120 TORs, participants maintained an ongoing on-path glance or towards a NDRT item.Participants' gaze at the time of the TOR was directed towards an NDRT item or on path for about the same number of observations (see Fig. 7, panel a).The response times to the TOR for the onset of a first glance to the instrument cluster, placing hands on the steering wheel, putting a foot on the accelerator pedal, and deactivating AD were generally longer for all repetitions when the participants looked at a NDRT item compared to when they looked on path (see Fig. 7, panels b-e).In line with this observation, a strong association can be observed between looking towards an NDRT item at the time of the TOR and the response times (see Table I).That is, all coefficients representing the deviation from the no-NDRT condition (i.e., β NDRT ) have a 95% HPD well above zero.In contrast, Fig. 7 does not reveal any consistent increase or decrease in response times due to repetition.In line with this observation, the association between the different exposures to TOR and the response times was not as strong as for the gaze direction at the TOR; in Table I only one 95% HPD does not include 0 (the hands-on-wheel response time at the fourth exposure).Also, this 95% HPD only marginally fails to include 0 (i.e., the lower boundary is 0).

C. Effect of Gaze Location at TOR on Response Times
Generally, the time to respond to a TOR is expected to be longer when a driver is looking towards an NDRT item than when a driver is looking on path (see Table II).Participants typically needed the shortest time to look towards the instrument cluster: the median response time was 0.7 s for On path and 1.3 s for NDRT.Furthermore, participants typically needed the longest time to place their foot on the accelerator pedal: the median response time was 3 s for On path and 5.3 s for NDRT.On average, participants needed shorter times to place the hands on the steering wheel (median = 1.5 s for On path, 3.2 s for NDRT) than to deactivate automation (median = 3.2 s for On path, 4.5 s for NDRT).Importantly, there is a large individual variation in the time needed for participants to respond to a TOR.In fact, the model suggests that, on average, 5% of participants need longer than 2.7 s to look to the instrument cluster, 5.2 s cluster than on path.Furthermore, in extreme cases, participants deactivated automation even before having looked on path.These observations are great safety concerns since drivers may not be sufficiently aware of the surroundings.Thus, in the design of future TOR procedures it is important to consider drivers' visual behavior along with take-over times (for an in-depth study on drivers' visual attention when responding to TORs in real traffic, see Pipkorn et al. [18]).As most previous studies did not look beyond take-over times, it is unclear whether a reduced visual attention to the forward road (on path) after a TOR occurred in other studies in real traffic (e.g., [16]) as well.Another way to minimize the risk that drivers might deactivate AD without knowing the reason behind the TOR (i.e., if they are looking towards the instrument cluster) is to include information about the TOR and the surroundings in the instrument cluster.

B. Association Between NDRT Engagement or Repeated Exposure and Drivers' Response to TOR
This study suggests that the association between gaze direction and the response process is stronger than the association between repeated exposure and the response process.Drivers' responses to a TOR typically take longer if a driver is looking towards an NDRT item compared to on path.In fact, looking towards an NDRT item prolongs not only the take-over time but also the times for: first glance to the instrument cluster (visual response) as well as placing the hands on the steering wheel and moving the foot to the accelerator pedal (motor responses).Thus, our findings are in line with previous findings on prolonged response times (mainly take-over times) when an NDRT item is present ( [5], [6], [16] compared to when it is not.Specifically, Naujoks et al. [16] reported median take-over times ranging from 2.71 to 4.90 s, depending on the type of NDRT that was being performed prior to the TOR.The time of 2.71 s is included in the 95% HPD for looking on path at the TOR, and 4.90 s is included in the 95% HPD for looking towards an NDRT item.Thus, our values are in line with previously observed response times in real traffic.An important difference between our study and the previous study by Naujoks et al. [16] is that we used the Bayesian framework for modeling and prediction, whereas they presented descriptive statistics on response times and performed inferential analyses within the frequentist framework.One of the advantages of the Bayesian framework over the more traditional frequentist framework is the preservation of uncertainty.Thus, we provide a model based on our sample data that not only predicts new data but also includes their uncertainty.This study presents not just one median response time, but a distribution of response times based on the model and the sample data.These probability distributions can be used by system designers to select values to match their priorities and understand the uncertainty in their decisions.For example, if a TOR is designed to be issued 7 s prior to exiting the ODD, we cannot be sure that all drivers will manage to deactivate automation in that time, as the 95% HPD for the 95th percentile driver who is looking towards an NDRT item when the TOR is issued ranges between 5.1 s and 8.0 s.Despite the advantages of the Bayesian framework, it is rarely used in the literature on human factors of automated driving, with a few exceptions (e.g., [26], [27], [28]).

C. Driver Monitoring in AD
One application of the results in this study is to inform driver monitoring system (DMS) design, to be included as part of future TOR strategies in AD.A DMS system could capture drivers that deactivate AD without being sufficiently back in the loop (if, for example, they deactivate AD without having looked on path).Ideally, a DMS could help preventing such deactivations from occurring as drivers may be late in detecting and responding to events occurring right after AD deactivation.For example, if DMS detects low visual attention levels on path, the vehicle could offer enhanced support (e.g., earlier warning in case of conflict) when possible, until the driver pays sufficient attention to the path.However, for cases when a driver's visual attention levels on path are too low, a better alternative is to redirect the driver's attention to the road before letting the driver deactivate the system.Furthermore, our findings suggest that drivers need longer to respond to TORs when looking towards an NDRT item than when looking on path.A DMS that can capture driver states, such as whether they are looking towards an NDRT item or on path, could be used to assure the safe transition of control.In this specific case, a driver looking towards an NDRT item would be predicted to need longer to glance towards the instrument cluster, place the hands on the steering wheel, deactivate automation, and place the foot on the accelerator.The safety of the transition of control could be assured in different ways.For instance, the median and 95th percentile response times presented in this study could serve as reference values for a typical driver's response times to the TOR, based on whether the driver is looking on path or towards an NDRT item.These reference values could then be used to detect deviating behaviors, such as drivers taking too long to place their hands on the steering wheel.In such cases, the AD would be able to intervene to avoid a potential crash by (for example) starting a minimum risk maneuver.
Aspects of drivers' visual behavior other than whether they look on or off path may be important to improve future DMS systems' ability to determine if a driver is available to respond to a TOR.Considering other aspects is especially important for identifying when drivers are directing their gaze to the forward path but are unable to respond to TORs due to, for example, impairments such as severe sleepiness, sudden sickness, cognitive distraction, and intoxication.Metrics that seem promising to capture driver impairments as part of a DMS strategy include eyelid movements and blinking patterns [29], [30], percent eye closure [31], and gaze entropy [32].

D. Limitations and Future Work
The findings presented in this study are based on a Wizard of Oz setup with a real vehicle, which provides a higher degree of realism than the driving-simulator and test-track studies reported in previous literature.However, the results may have been influenced by the presence of the test leader and wizard driver in the vehicle.In particular, the fact that all drivers responded to the TOR within 10 s may be partly due to participants' desire to perform well when they are part of a study.Moreover, they were Volvo car employees in the Gothenburg area; although they were not involved in work related to AD product development, they may not be representative of an international (or even Swedish) population.Thus, the predicted response times presented in this paper should be used with some caution; while they represent the current state of knowledge given the observed data, they may change as more data is collected and added to the distribution (in line with the Bayesian philosophy).As all drivers in the study responded to the TOR, the important question-What happens if a driver does not (voluntarily or involuntarily) respond to a TOR? -remains unknown.However, it seems reasonable for future AD to always incorporate a safety backup in case a driver does not respond.In this study, the Wizard driver served as the safety backup.Future work should continue to investigate how DMS could be used to capture driver behaviors, which may prevent drivers to safely respond to TORs (e.g., sleeping), so that the AD act appropriately.Overall, this study focused on TOR responses comprising observable behaviors that can be measured using cameras.A detailed understanding of drivers' cognitive state while responding to TORs remains unknown.Future studies should further investigate the possibility of measuring drivers' cognitive state using DMS.For example, DMS systems could capture gaze concentration, which could be an indication of cognitive distraction [33].In addition, note that this paper did not perform a thorough model-selection procedure before using a log-normal distribution.Thus, other distribution types (e.g., Gamma) may give slightly different predictions about driver response times to TORs, especially for the 95th percentile values.Finally, this study did not perform detailed analyses on how the type of NDRT item or the level of NDRT engagement influenced the response times.This study presents response time distributions based on data from situations in which participants were free to engage in NDRTs of their own choice.Thus, this study does not separate the influence of different NDRT items or different levels of NDRT engagement on response times.

V. CONCLUSION
In real traffic, drivers can respond to TORs and resume manual driving within ten seconds.The response constitutes a sequence of actions including looking towards the instrument cluster and on path, placing the hands on the steering wheel and the foot on the accelerator pedal, and deactivating automation.Importantly, both the timing and the order of the actions vary across individuals.However, drivers always look towards the instrument cluster before placing their hands on the steering wheel and deactivating automation.Despite the importance of on-road glances for traffic safety, it is not necessarily the case that drivers always start their response by looking on path.In fact, when drivers are looking off path at the time of the TOR, they are more likely to respond to the TOR by looking to the instrument cluster before looking on path.Importantly, for 5% of the TORs, drivers deactivated automation even before having looked on path.Note that, previous research focusing on driver visual attention levels indicates that it may even take longer than 10 s (∼15 s) before all drivers show the same level of visual attention on path as in manual driving [18].The time needed for the response process is prolonged if the driver is looking towards an NDRT item rather than on path at the time of the TOR.Drivers' gaze direction at the time of the TOR (on path or towards NDRT item) is more important for predicting the response times than is repeated exposure to the TOR.Therefore, this study presents predicted response times for a driver who is looking towards an NDRT item or towards the forward roadway at the time of the TOR.These response times can be used as reference values so that future DMSs will be able to capture potentially deviating driver behaviors, such as taking too long to place their hands on the steering wheel, indicating that the driver is unlikely to deactivate automation in a timely manner.

Fig. 2 .
Fig. 2. Instrument cluster view for a TOR (top), and when the two steering wheel buttons are being pressed to deactivate AD, showing the turquoise bars moving toward each other (bottom).In both views, the information "Hold steering wheel buttons to take control" is displayed.

Fig. 3 .
Fig. 3. Participants completed a 60-min continuous drive (two 30-min laps).Each lap started with manual driving, followed by AD and a practice TOR.Each of the four test AD drive segments ended with a TOR (TORs 1-4).

Fig. 4 .
Fig. 4. Video views used for manual coding show: (a) driver's upper body and hand placement, (b) driver's feet and brake/accelerator pedals, (c) driver's face and eyes, and (d) the forward roadway.

Fig. 7 .
Fig. 7. Frequency of events in which participants looked on path and towards an NDRT item at the time of the TOR (Panel a).Response times to the TOR for (b) first glance to the instrument cluster, (c) hands on wheel, (d) foot to accelerator pedal, and (e) automation deactivation, across the four repetitions and TOR gaze direction.The number of datapoints included for each repetition and gaze direction is displayed at the bottom of each panel.