In-Vehicle Human Machine Interface: Investigating the Effects of Tactile Displays on Information Presentation in Automated Vehicles

Background: Semi-autonomous vehicles still require human drivers to take over when the automated systems can no longer perform the driving task. Objective: The goal of this study was to design and test the effects of six meaningful tactile signal types, representing six driving scenarios (i.e., navigation, speed, surrounding vehicles, over the speed limit, headway reductions, and pedestrian status) respectively, and two pattern durations (lower and higher urgencies), on drivers’ perception and performance during automated driving. Methods: Sixteen volunteers participated in an experiment utilizing a medium-fidelity driving simulator presenting vibrotactile signals via 20 tactors embedded in the seat back, pan, and belt. Participants completed four separate driving sessions with 30 tactile signals presented randomly throughout each drive. Reaction times (RT), interpretation accuracy, and subjective ratings were measured. Results: Results illustrated shorter RTs and higher intuitive ratings for higher urgency patterns than lower urgency patterns. Pedestrian status and headway reduction signals were associated with shorter RTs and increased confidence ratings, compared to other tactile signal types. Lastly, among six tactile signals, surrounding vehicle and navigation signal types had the highest interpretation accuracy. Conclusion: These results will be used as preliminary data for future studies that aim to investigate the effects of meaningful tactile displays on automated vehicle takeover performance in complex situations (e.g., urban areas) where actual takeovers are required. The findings of this study will inform the design of next-generation in-vehicle human-machine interfaces.

fully autonomous vehicles (SAE Level 5) [2], current automa-23 tion technology, such as SAE Level 3 automation, is not 24 perfect. For example, SAE Level 3 automation may fail to 25 The associate editor coordinating the review of this manuscript and approving it for publication was Xiaojie Su .
perform the driving task in many driving conditions (e.g., 26 encountering erased lane markings or in poor visibility), 27 which would prompt the vehicle to abruptly request the driver 28 to manually take over control of the vehicle in a limited 29 matter of time [2]. This two-phase (signal response and post-30 takeover), three-step takeover process ( Fig. 1) entails first 31 perceiving and processing the takeover request and then need-32 ing to quickly shift their attention while becoming aware of 33 their surroundings and assessing the situation, then moving 34 their hands and feet back to the driving position to manually 35 sensory channels. For example, drivers need to reorient (pay 48 attention to the road) and regain situation awareness during 49 the takeover process [6], [9], [10]. However, a takeover in resources theory [11] suggests that tactile displays may be a 63 good option as the tactile modality may be more available 64 than visual and auditory modalities in a data-rich driving 65 environment. 66 Previous research has demonstrated the benefits of tactile 67 displays as an assistive HMI in a large body of research, 68 which has shown that tactile displays significantly improved 69 decision making with faster cognitive processing/response 70 speeds, while reducing cognitive workload [4], [8], [12], [13], 71 [14], [15], [16], [17], improving situation awareness [18], 72 and enhancing vehicle handling [19], [20]. For example, a 73 study conducted by Van Erp and Van Veen [13]  the position of the automated vehicle, to assist drivers in sens-85 ing failures in vehicle automation while engaging in NDRTs. 86 It found that notifications presenting spatial information on 87 surrounding traffic required fewer mental resources, which 88 allowed participants to interpret sensing failures in vehicle 89 automation with higher accuracy and lower mental workload. 90 In addition, Telpaz, Rhindress, Zelman, and Tsimhoni [19] 91 studied a haptic seat that presented spatial information of 92 approaching vehicles and found that participants who had a 93 haptic seat showed shorter reaction times in scenarios requir-94 ing lane changes than participants with no haptic seat.

95
Given the advantages of tactile cueing, researchers started 96 focusing on changing characteristics (e.g., rhythm, duration, 97 intensity) to create meaningful tactile patterns to repre-98 sent complex driving scenarios during a takeover. A recent 99 review article summarized studies that used tactile displays 100 as the HMI in automated vehicles and categorized the stud-101 ies into either instructional signals (i.e., instructions for 102 drivers to maneuver their own vehicles) or informative signals 103 (i.e., representing spatial location/status of approaching vehi-104 cles/pedestrians/obstacles in the environment) [21]. Exam-105 ples of instructional signals include navigational [12], [13] [14], headway reductions [4], [8], [15], [16], [17] [16] ran a study that compared tactile, visual, 113 and auditory warnings for rear-end collision prevention (i.e., 114 informative signals) during a simulated drive, using a higher 115 urgency pattern that had 200 ms, with an 800 ms pause per 116 second. They found that drivers with a tactile warning had 117 significantly shorter response times than drivers without a 118 warning or drivers with visual warnings. Here, the tactile 119 display was conveyed via three tactors fastened on a waist 120 belt and positioned on the driver's abdomen to simulate a 121 driver's seatbelt. Moreover, a study conducted by Chang,122 Hwang, and Ji [14] compared tactile, visual, auditory, and 123 multimodal displays during a simulated drive, which gave 124 navigational information (i.e., instructional signals) such as 125 left, right, and straight, along with a speed limit warning. 126 This study conveyed tactile warning signals via three types 127 of patterns: 1) 12 tactors attached to the driver's seat were 128 activated in sequential bursts of 120 ms with a 510 ms 129 pause from back to front to represent ''proceed straight'', 2) 130 one tactor on both the left and right sides of the seat were 131 attached to represent the ''go left'' or ''go right'' signals and 132 presented 158 ms bursts with a pause of 46 ms, and 3) four 133 tactors were placed on the seat back to represent the speed 134 limit warning presenting two 726 ms bursts with a pause of 135 78 ms. The study found faster response times for the tactile 136 and multimodal displays in addition to higher satisfaction 137 and lower subjective workload for participants who had a 138 tactile display versus an auditory or visual one. However, 139 these papers only used one type of signal and/or pattern 140 for only one information presentation purpose (i.e., either 141 instructional or informative, but not both). In other words, dif-142 ferences in performance under the effects of meaningful tac-143 tile signal type and pattern (in different perceived urgencies) 144 have not been extensively studied. Here, perceived urgencies 145 were manipulated by varying signal durations and interstim-146 ulus intervals (i.e., pause periods between bursts) [6], [36], 147 [37], [38]. Thus, it is still unclear whether multiple mean-148 ingful or complex tactile patterns can be used altogether 149 (i.e., only activated for corresponding driving scenarios) to 150 communicate the needs of takeover and convey more infor-151 mation to help the takeover task and be reliably and intuitively 152 identified by drivers.

153
Therefore, the goal of this study was to design and test the 154 effects of signal types representing six most representative 155 driving scenarios, i.e., navigation, speed, location/status of 156 surrounding vehicles, over the speed limit, headway reduc-157 tions, and pedestrian status, based on previous studies (e.g.,

191
There were six signal types (Table 1)  The experiment employed a 6 (signal type: navigation, 199 speed, surrounding vehicle, over speed limit, headway reduc-200 tion, pedestrian status) × 2 (pattern: lower urgency, higher 201 urgency) full factorial design. Here, signal type and patterns 202 were within-subject factors. The six tactile signal types, with 203 subcategories, were presented in three locations, i.e., seat 204 back, pan, and/or belt. The six signal types represent the 205 most common takeover scenarios from the literature (see a 206 review, [21]). These signals were designed based on other 207 studies, including the tactile locations, the number of tactors, 208 and vibration intensity and sequence used (see an exam-209 ple design guideline, [36]), as well as a few iterations of 210 in-lab prototype testing. For example, navigational signals 211 had three subcategories, left turn (presented on either belt 212 or back), right turn (presented on either belt or back), and 213 U-turn (presented on either belt or pan). We intentionally 214 presented signals at different locations to eliminate potential 215 location effects, given that previous studies played tactile 216 signals at different locations (e.g., seat back [12], [40], pan 217 [13], [22], [24], and belt [23]) for the same meanings, but 218 the comparisons of locations on takeover performance have 219 not been widely studied. Similarly, speed signals had two 220 subcategories, speed up (presented on either belt or pan) 221 and slow down (presented on either back or pan). Surround-222 ing vehicles approaching signals had three subcategories, 223 approaching from the left side (presented on either back 224 or pan), approaching from behind (presented on back), and 225 approaching from the right side (presented on either back or 226 pan). Both over speed limit and headway reduction signals 227 only had one subcategory to represent speeding (presented 228 on either back or pan) and forward collision (presented on 229 belt), respectively. Finally, pedestrian status warning signals 230 had two subcategories played on the seat belt: traveling left-231 to-right or right-to-left. Each tactile signal was presented 232 in two patterns: lower and higher urgency. Lower urgency 233  of 107.5 ms. See Table 1 for a summary of signals and patterns 240 that were designed and examined. Once participants pressed the button, they needed to state 274 their interpretation of the signal, and then rate their confi-275 dence in their answer and intuitiveness of the tactile signals 276 both on a scale of 1(low) to 5 (high). The interpretation 277 accuracy was also recorded. The study lasted about two 278 hours and was split into four sections to help prevent fatigue. 279 At the end of the experiment, participants filled out a post-280 experiment questionnaire, which asked questions about their 281 overall experience, and were then debriefed.

283
The dependent variables were put into three categories: 284 a) reaction time (in milliseconds (ms)), which was the time 285 between the onset of the tactile signal and the moment the par-286 ticipant pressed the button on the dashboard; b) interpretation 287 accuracy, participants were presented with 120 tactile signals 288 (20 signals as shown in Table 1, each randomly repeated three 289 times in two urgency patterns) and were asked to provide 290 an answer after each signal, as to what they felt the tac-291 tile signal was communicating, which measured the number 292 of correct answers in each of the 12 conditions (6 signal 293 types × 2 patterns), and c) subjective satisfaction ratings, 294 which were participants' ratings based on the confidence in 295 their answers and intuitiveness of the tactile signals, both on 296 a 5-point rating scale (1 low -5 high).

298
Dependent variables were analyzed using a two-way repeated 299 measures analysis of variance (ANOVA) with tactile sig-300 nal and pattern as independent variables. For violations of 301 sphericity tests, degrees of freedom were corrected using 302 Greenhouse-Geisser estimates. Bonferroni corrections were 303 applied for multiple comparisons to identify significant dif-304 ferences and interactions between each level. The statistical 305 analysis was conducted using IBM SPSS Statistics 28.0 and 306 evaluated at a significance level of p < 0.05. Effect size was 307 presented as partial eta squared (η 2 p ).

309
A. REACTION TIME

437
Regarding accuracy, navigation and surrounding vehicle 438 signals had the highest accuracy compared to the other sig-439 nal types. These results are in accordance with previous 440 research that used vibrotactile warning signals to represent 441 spatial/navigational directions in the driving environment 442 during semi-autonomous driving, leading to increased accu-443 racy of information interpretation of directional signals [12], 444 [19], [20], [24], [39], [44]. One likely explanation may be 445 that drivers have become more familiar with navigation and 446 surrounding vehicle signals in their day-to-day driving expe-447 rience (e.g., blind spot warnings as surrounding vehicles 448 signal; and GPS/mobile apps for navigation purposes). Even 449 though these signals are generally applied via different sen-450 sory channels (i.e., visual and auditory), they may be more 451 capable of processing the same type of information [11]. 452 Follow-up studies may conduct, for example, semi-structured 453 interviews or focus groups to gain more insights in this regard. 454

455
As described in Methods, higher urgency patterns consist 456 of two shorter warning signals, while lower urgency pat-457 terns are longer warning signals. Overall, the higher urgency 458 pattern significantly reduced reaction time, by 359.4 ms on 459 average, and had higher intuitive ratings, compared to lower 460 urgency patterns. This finding is consistent with previous 461 research in that participants tended to prefer signals (i.e., 462 measured subjectively) with shorter ISI durations compared 463 to longer ISI durations [6], [36], [45], [46]. For example, 464 Pratt et al. [45] investigated whether scalable levels of per-465 ceived urgency could be achieved utilizing tactile signals 466 by measuring the changes between the vibrotactile pulse 467 rate and its relationship to perceived urgency and annoyance 468 ratings. That study found that faster pulse rates (shorter ISI) 469 resulted in signals being perceived as having higher urgency. 470 In our study, we also found that shorter burst durations 471 and ISI durations were correlated with faster reaction times 472 (with objective data). One possible explanation for this result 473 could be that the shorter burst and ISI durations create a 474 sense of urgency [6], [37], [38], [45], [46], which helped 475 drivers quickly process and comprehend the signal informa-476 tion (measured by reaction times). Alternatively, the signal 477 duration could be the cause of the differences between the 478 higher and lower intensities. In our design, the higher urgency 479 patterns were shorter in overall duration time compared to the 480 lower urgency patterns. The shortened time duration allowed 481 drivers to start processing the signal meaning and return their 482 attention to the driving environment earlier, thus reacting 483 faster to distal stimuli.

485
There are a few limitations of this study. First, participants in 486 experienced a total of 120 signals under six warning signal 487 types and two different types of patterns. Although our goal 488 was to compare the tactile signals and patterns, and we gave 489 shown that demographic information such as age or gender 511 may cause individual differences in task performance. For 512 example, older adults who may be experiencing cognitive and 513 psychomotor declines may have slower and more variable 514 reaction times compared to younger adults [47], [48], [49]. The authors would like to thank bachelor's student Brenna 546 Nettles-Miller for assisting with data collection, and Collin 547 Li for interface development.