Effects of Display Response Latency on Brain Activity During Device Operation

Human–machine interface (HMI) devices have become multifunctional and complicated to operate, requiring better usability. Nevertheless, response latency (time lag between operation input and display output) negatively impacts device usability because it is stressful and reduces the operator’s sense of agency (SoA)—the sense that the operator is the one who is causing or generating an action. Stress and SoA information can be obtained by measuring physiological indicators, such as brain activity. This study was designed to investigate the effects of response latency on the operator’s brain activity state through near-infrared spectroscopy, which measures the brain activity state based on changes in the concentration of oxygenated hemoglobin ( $\Delta $ Oxy–Hb). In this study, 15 subjects performed a target-tracking task in which they rotated an HMI commander used as an in-vehicle device to manipulate LED markers on a display. The temporal response latency (0, 50, 100, and 150 ms) between manipulating the commander and the display response was introduced, and the subject’s brain activity, subjective evaluation, and performance efficiency were measured. The results revealed that brain activity differed depending on whether the latency was recognized rather than the latency length, and that $\Delta $ Oxy–Hb at the anterior prefrontal cortex of those who recognized the latency increased significantly, whereas that of those who did not recognize the latency did not increase. Specifically, latency perception activates the prefrontal cortex due to stress and the inability to prepare normally for the next action, and not recognizing latency inactivates the premotor cortex.


I. INTRODUCTION
In recent years, innovative technologies have been proposed in the development of multifunctional human-machine interface (HMI) devices, including touch panels and dial commanders in smartphones and automotive navigation/control interfaces. However, since users check the screen display The associate editor coordinating the review of this manuscript and approving it for publication was Gang Wang . response to input the next command, a significant time lag between the command input and the screen display output (hereafter referred to as ''response latency'') is reported to be bothersome for users [1] and a major bottleneck for usability [2]. In other words, long response latency prevents intuitive operation and frustrates the system user. Therefore, to contribute to breakthroughs in HMI device development, studies to evaluate the impact of response latency on HMI operation are needed.
Previously, numerous studies have focused on task performance efficiency, such as the time required to complete a task [3], [4] and task error rate [5], [6], in which response latency is intentionally inserted. In addition, subjective evaluations have been conducted to examine subjects' recognition of response latency [7], [8] or their sense of agency (SoA)-the sense that the operator is causing or generating an action [9], [10], [11] under latency-inserted environments. Nevertheless, a well-known problem with studies that have evaluated the effect of response latency in terms of performance efficiency is that they do not consider the operator's stress and SoA state. Furthermore, subjective evaluations are less quantifiable, and their relationship with human physiological and physical responses remains unclear.
To address these challenges, measuring biomarkers such as brain activity or amylase activity state has been applied in usability studies in recent years. For example, the prefrontal cortex activity [12], [13] and amylase activity in saliva [14], [15] have been used to assess stress, while the premotor cortex has been reported to be related to the SoA [16], [17]. By investigating the effects of response latency on brain activity, we will explain the relationship between response latency and operator stress and SoA.
Therefore, this study was designed to investigate the effect of display response latency on stress and the SoA state based on biomarkers such as brain activity and amylase activity, as well as performance efficiency and subjective evaluation during the rotary commander manipulation tracking task. Furthermore, by examining the difference in brain activity between the groups that recognized the response latency and those that did not, we will evaluate the effect of latency recognition and the accompanying stress on brain activity, and we will also discuss the brain response to latency.

II. RELATED WORK
Since HMI users perform input operations based on visual/sensory feedback, evaluating the impact of display latency on input operations is needed; accordingly, various studies on this issue have been conducted to date. Therefore, studies have been reported on not only visual feedback delay [18], [19], which is the time lag between an input operation and the display response, but also auditory feedback delay [20], which is the time lag between an input operation and the sound, and tactile feedback delay [21], which is the time lag between when a device touches an object and when force is transmitted to the operator's side, such as in telesurgery.
While most studies investigating the effects of visual feedback delay have focused on subjective ratings and performance efficiency, few studies have focused on biomarkers. However, examining biomarkers is quite effective because it can confirm human responses with little cognitive bias and allows a quantitative sense of operation evaluation. For example, cognitive stress is assessed with prefrontal cortex activity [12], [13], and mental or physical stress is assessed with amylase activity [14], [15]. In addition, it has been determined that motor cortex activity is associated with an SoA [16], [17], which improves usability [22]. Therefore, measuring brain activity and amylase activity under latency-inserted environments is effective in assessing the cognitive stress and SoA loss caused by latency.
Currently, electroencephalogram (EEG) [23], positron emission tomography (PET) [24], magnetic resonance imaging (MRI) [25], and NIRS are used in neuroimaging methods, among which NIRS is suitable for measuring tasks involving motion and tasks that mimic daily life because of its low noise and low-constraint measurement techniques [26]. NIRS is a technique for measuring blood flow dynamics in the cerebral cortex, which is involved in cognitive function, using nearinfrared light.
Some studies that have examined the effect of latency on brain activity have focused on tasks that involve judging whether a hand projected on a screen is the participant's hand [27], [28], but few have focused on devices used FIGURE 2. Experimental protocol and measurement timing of each indicator. In task time, participants were required to operate the HMI commander while watching the LED on the screen. In rest time, participants took their hand off the HMI commander. After a 5-minute break, the experiment was conducted under the following condition.
in real life. Therefore, this study focuses on devices used in daily life to investigate the effect of latency on brain activity.
This study will investigate the effect of visual feedback delay on the operation of HMI devices used in automotive navigation/control interfaces by measuring brain activity using NIRS.

III. EXPERIMENTAL STRATEGY
Fifteen right-handed healthy men aged 20-24 participated in this study. The experimental task was a target-tracking task by rotating a rotary commander. The target was displayed on a display, and LED markers, hereafter referred to as ''LED'', that the subjects moved were displayed next to the target (Fig. 1). While gazing at the target, the subject rotated the commander so that the LED he controlled was next to the target. The next target was automatically displayed when the LED reached the target position. As the commander was rotated clockwise and counterclockwise, the LED moved upward and downward, respectively, and when the LED reached the target position, the next target was automatically displayed. The commander was designed to produce a click sensation every 12 deg. of rotation using a mechanism to generate a click feeling; the torque required to produce the click feeling was approximately 0.20 Nm. The click angle was set in accordance with off-the-shelf products from Mazda Motor Corporation, which are used as an actual automotive navigation/control interface HMI controller. As shown in Fig. 2, subjects repeated the task for 30 seconds and rested for 30 seconds three times in each condition following the experimental protocol.
The experimental conditions were the response latency from the commander rotation to the LED position change, and four conditions were selected: Condition 1 with no response latency and Conditions 2, 3 and 4 with response latencies of 50 ms, 100 ms, and 150 ms, respectively. The response latencies for each experimental condition were determined based on the results of a preliminary experiment. In a preliminary experiment, each of the above four latencies was tested 5 times to ensure that the 15 participants were recognized. Fig. 3 shows the average values and the results of Tukey's HSD test (null hypothesis: the mean of the perceived rate) between conditions. Significant differences were found between all conditions except for 0 ms and 50 ms. Since the response latency of 150 ms was recognized by almost all subjects, a longer latency was not included in this study. The latency inherent to the equipment in this experiment was less than 1 ms, which is quite short, so it was assumed to have no effect on the human response or operation. Subjects held the commander and started manipulating it at the start of task time and released it at the end of task time. The order of the inserted latencies was randomized to eliminate order effects. All the instructions regarding the experiment were provided verbally. To prevent any differences in the subjects' proficiency in operating the controller from affecting their brain activity, they practiced the operation in advance of the experiment using the same equipment.
The angle information was obtained from the rotary encoder interlocked with the commander, and the lighting color of the two-color LED was controlled by Arduino Due. A latency was inserted by Arduino Due software between the angle information input from the rotary encoder and the change in LED color.
The subject's brain activity was measured using an NIRS optical topography system (ETG-7100; Hitachi Medical Corporation, Tokyo, Japan). Fig. 4 shows the positions of the brain function area and measurement channel. The NIRS measurement probe was attached based on the international 10-20 system [29] for electrode placement, which is widely used for site identification. The motor and somatosensory cortices were measured: 24 ch in the right and left hemispheres (Probe1) (Probe2), respectively, with Cz at the center and in the parietal head at 22 Ch (Probe4), while the prefrontal cortex was placed at 22 Ch from Fp1 and Fp2 backward (Probe5), for a total of 92 Ch. The detection range was 25-30 mm depth into the cortex under the scalp, and the spatial resolution was 30 mm. The sampling frequency was 10 Hz, which enabled real-time measurement. During the experiment, to ensure that the subjects were not affected by the external environment, we minimized any potential distractions in the room from sounds, smells, or sights. As shown in Fig. 2, amylase activity was measured once before and after each condition using an amylase monitor and chip (Nipro, Osaka, Japan). Subjects were instructed not to drink alcohol on the previous day, not to eat for 2 hours before the trial, and not to drink water 10 minutes before the trial. This study was conducted after obtaining approval from the Ethical Review Committee for Research on Human Subjects of Doshisha University.

IV. ANALYSIS METHOD A. SUBJECTIVE EVALUATION
To evaluate the effect of differences in response latency on the operational sensation, a subjective evaluation was conducted after each condition using the semantic differential scale method (SD method). The evaluation item was ''ease of target tracking'' and was rated on a 7-point scale from +3 to -3 (+3: very good, -3: very bad). The results of the subjective evaluation were analyzed using Tukey's HSD test at a 5% level of significance (null hypothesis: the means of the subjective evaluation scores are equal in each condition). Here, Tukey's HSD test is robust for homogeneity of variance [30] and normality [31]; thus, a detailed discussion is omitted.

B. NUMBER OF CLICKS
To evaluate the effect of differences in response latency on performance efficiency, the number of clicks was calculated from the number of times the target was tracked during task time. Using these values, we performed Tukey's HSD test (null hypothesis: the mean number of clicks is equal across conditions) at a significance level of 5% across conditions.

C. AMYLASE DATA
To evaluate the effect of the difference in response latency on amylase activity, the concentration of amylase in saliva was measured. The amylase concentration in saliva was measured before and after the trial, and the rate of change [%] was calculated using the following equation.
where SSA 0 and SSA are the amylase activity values [kIU/l] before and after the trial, respectively. We averaged the responses of the subjects to each stimulus and compared them between conditions. Using the results, Tukey's HSD test (null hypothesis: the means of the percent change in amylase concentration between conditions are equal) was conducted at the 5% level of significance to investigate whether there were differences between conditions.

D. NIRS DATA
As increases in brain activity are accompanied by an increase in oxygen consumption and a corresponding increase in blood flow [32], NIRS often detects increases in the concentration of oxygenated hemoglobin (Oxy-Hb) and total-Hb and decreases in the concentration of deoxygenated hemoglobin (Deoxy-Hb). In particular, Oxy-Hb is considered to be the parameter that best reflects brain activity [33]. When using an optical topography system, the head is irradiated by near-infrared light from an irradiation probe, which is reflected by hemoglobin in the cerebral blood vessels in the cerebral cortex. The reflected light that is incident on the light-receiving probe is used to calculate the change in hemoglobin concentration in the blood [34]. The approximate straight line connecting the values measured during the before and after task time was set as the baseline for the analysis, and an integral analysis was performed. The data obtained from an NIRS measurement were the product of the change in Hb concentration and the optical path length. Since the optical path length varied depending on the measurement channel and the subject, it was not possible to directly compare the activation states between subjects and sites. We followed the protocol of previous studies utilizing NIRS and addressed this issue by standardizing the measurement data [35]. The standardized data were tested at a significance level of p = 0.05 to identify the areas of significant cortical activity. A ttest was performed for the brain activity in each condition (null hypothesis: the average values of Oxy-Hb concentration during the rest and task times are equal). Tukey's HSD test was performed to compare brain activity between experimental conditions (null hypothesis: the average values of Oxy-Hb concentration in the task time between each condition are equal). All analyses were performed using MATLAB R2019a.

V. EXPERIMENTAL RESULTS AND CONSIDERATION
A. SUBJECTIVE EVALUATION Fig. 5 shows the average subjective evaluation score for each condition. The mean value for ''ease of adjusting LED'' was higher in the short-latency condition and lower in the long latency condition. The comparisons between conditions showed that Conditions 1 and 2 were significantly higher than Conditions 3 and 4. This indicates that longer latency makes the task more difficult to perform and causes discomfort. This may be due to SoA loss caused by the extended latency time. In previous studies [36], [37], there were a number of reports on SoA loss due to latency insertion, and the results of this study confirm these findings.    and Conditions 2-4. The results indicate that the subjects' performance efficiency worsens as latency increases.
C. AMYLASE DATA Fig. 7 shows the average rate of change in amylase activity before and after task time for each condition. In the average rate of change, there was a tendency for amylase activity to increase in conditions with longer latency. However, the standard deviation was very large (330% of the mean in Condition 4), and Tukey's HSD test results showed no significant differences between conditions. Therefore, the effect of latency on amylase activity was not confirmed and stress evaluation by amylase activity was not possible in this study. Fig. 8 shows the brain activation map based on Oxy-Hb. The color and value of the bar in the lower part of Fig. 8 correspond to the t-value of the t-test. The color and numerical values of the color bar shown in the lower part of the figure correspond to the t-value of the t-test. When the change in oxygenated hemoglobin concentration increases during the trial compared to the resting state, the t-value is a positive value, and the color on the color bar indicates the red side. In contrast, when the brain is inactive during the trial compared to the resting state, the blue side of the color bar is shown. In this study, the areas that showed values higher than 2.15 (5% level of significance), which is the rejection limit with 14 degrees of freedom (15 subjects), showed significant activation compared to the rest time. As shown in Fig. 8, activation of the medial premotor area was confirmed in Condition 1, where there was no latency. In Condition 4, which had the longest latency of 150 ms, activation in the anterior portion of the prefrontal cortex was observed. The following is a discussion of these two regions.

D. NIRS DATA
Although the premotor cortex can be divided into dorsal and ventral parts according to its functional characteristics, the dorsal part (dorsal premotor cortex: PMd) is reported to be involved in motor task control and learning and is important for adaptation to complex tasks [38]. In addition, the anterior part of the motor cortex has been reported to be associated with a SoA [16], [17]. The results show that in Condition 1, where there was no latency, the subjects were able to perform the task while maintaining a SoA, while in the condition with latency, the activation of the same area was not observed, suggesting that the SoA was lost due to the latency. In a previous study, it was reported that SoA loss was observed when the feedback differed from the participant's sense of agency [39]. Therefore, in Condition 1, the participants were able to maintain their SoA in performing the task without latency.
In contrast, the anterior prefrontal cortex is sensitive to stress responses [40], [41] and is considered to be the area most susceptible to the effects of stress exposure [42]. In previous stress assessment studies using NIRS, activation was confirmed in a variety of tasks ranging from mental stress, such as a mental arithmetic task [12], [13], to physical stress, such as an airplane flight simulation [43].
Correlations between other indicators and brain activity were also examined. The NIRS data from the left front of the prefrontal cortex showed a significant correlation with performance efficiency (p = 0.033), however, no correlation was found between brain activity and subjective evaluation.

E. RELATIONSHIP BETWEEN LATENCY RECOGNITION AND BRAIN ACTIVITY
In the preliminary experiment, each subject was given five trials for each response time to examine the percentage of latency recognition, so that the recognition rate was 100% if the subject recognized all five trials and 0% if the subject did not recognize any of the trials. As shown in Fig. 3, the results of a preliminary experiment examining the perception of latency for each condition showed that the average perceived latency was 2.7% and 9.3% for the condition with no latency and 50 ms latency inserted, respectively. Here, in 2.7% of VOLUME 11, 2023 FIGURE 9. Topographical map of Oxy-Hb of nonrecognition group and recognition group. Red indicates high levels of Oxy-Hb, and blue indicates less activation. Oxy-Hb on the left side of the prefrontal cortex and premotor cortex in the recognition group is higher than that in nonrecognition group. α R and α N refer to the critical value of the recognition group and nonrecognition group, respectively. the trials where there was no actual latency, the subjects falsely perceived a latency. Thus, in both conditions, subjects answered that they did not perceive any latency at a rate of more than 90%. The average latency recognition rate was 96% for the 150 ms latency, confirming that almost all subjects recognized the latency. Preliminary experimental results showed that there were no significant individual differences in these three latency times. However, in the 100 ms latency trials, the mean recognition rate was 53% with a standard deviation of 38%, indicating larger individual differences in recognition rates than in the 0, 50, and 150 ms response latency trials. In the 100 ms latency trials, there were both subjects who recognized 100% of the latency and subjects who did not recognize latency at all. Based on these results, we divided the participants into two groups: the latency nonrecognition group, which had a less than 50% recognition rate for a 100 ms latency in the preliminary experiment, and the latency recognition group, which had a more than 50% recognition rate for a 100 ms latency. We then investigated how the brain activity in the two groups differed in the 100 ms latency-insertion condition. Here, the latency nonrecognition group and the latency recognition group had 7 and 8 participants, respectively. The brain activity maps of the two groups are shown in Fig. 9. As shown in Fig. 9, the brain activity states of the nonrecognition group and the recognition group were different despite the same latency time. The brains of the recognition group tended to be more active than those of the nonrecognition group, especially in the prefrontal cortex and the medial part of the premotor cortex.
The prefrontal cortex is involved in the stress response as described in Section V-D, and while no significant activation was observed in the nonrecognition group, significant FIGURE 10. Comparison of subjective evaluation results between the latency nonrecognition group and the latency recognition group. As a result of t -test, no significant difference was confirmed between groups.

FIGURE 11.
Comparison of performance efficiency results between the latency nonrecognition group and the latency recognition group. As a result of t -test, no significant difference was confirmed between groups. activation was observed in the recognition group. This indicates that the recognition group felt stressed by the latency detection. This is because the subjects were instructed to perform the task as quickly as possible before the start of the experiment; thus, the recognition group recognized the latency and felt that they were disrupted from performing the task. In contrast, the nonrecognition group did not recognize the latency; thus, they were able to perform the task without feeling stressed.
In addition to being involved in generating the SoA, it has recently been determined that the premotor cortex integrates visual and somatosensory information during movement to plan and prepare for subsequent actions [44] and to calculate the discrepancy between prediction and the actual movement [45]. In this study, the participants integrated the somatosensory information of their hands rotating the commander with the visual information of the LED moving on the display. In the latency-insertion conditions, the recognition group showed activation in the same area, suggesting that they were able to recognize the gap between their predicted LED movement and the actual LED movement, calculate the discrepancy, and integrate visual and somatosensory information based on this information. In contrast, the nonrecognition group failed to recognize the latency, thus resulting in the inactivation of the same area due to the inability to integrate visual and somatosensory information. Considering the results of Section V-D, in the short-latency (e.g., less than 50 ms) conditions, all subjects were able to integrate visual and somatosensory information, whereas in the long latency (e.g., more than 100 ms) conditions, only those subjects who were able to calculate the discrepancy between their predicted LED motion and the actual LED motion integrated both types of information, and those who were not able to calculate the discrepancy did not. From these results, it is determined that the recognition group felt stress by recognizing the latency, and thus, the prefrontal cortex was activated, and the premotor cortex was activated because visual and somatosensory information was integrated by calculating the latency in the brain, whereas the nonrecognition group did not recognize the latency; thus, these areas were not activated. These results indicate that brain activity differs even under the same latency conditions depending on differences in latency recognition.
In addition, the prefrontal cortex was significantly more activated in the recognition group than in the nonrecognition group, indicating that stress assessment can be performed by the activity state of this region, which is sensitive to stress responses. In previous studies, amylase activity was reported to be activated by social stress, such as TSST [14], and stress related to fear and exercise, such as surgery [46] and cycling [47]. However, in this study, stress detection by amylase activity could not be performed, and no significant difference was found in the comparison between the nonrecognition group and the recognition group. This suggests that the stress assessment ability of NIRS-based prefrontal brain activity measurements is superior to that of amylase activity in assessing stress associated with latency. Here, whether the results of subjective evaluation, work efficiency, and amylase activity differed between the delayed noncognition group and the cognition group was examined by t-tests with a significance level of 5%. As a result, no significant difference was confirmed in any index under all conditions (Figs. 10, 11 and 12). Although no differences between the two groups were observed in subjective evaluation, number of clicks, or amylase activity in all conditions, the finding of differences in brain activity alone indicates that NIRS detected differences between the two groups that could not be detected by efficiency or subjective evaluation. It is expected that the findings of this study, obtained by quantitatively evaluating the SoA and stress, which had been evaluated qualitatively in the past, considering individual differences, will be applied in HMI development.

VI. CONCLUSION
In this study, we quantitatively evaluated brain activity, subjective information, performance efficiency, and amylase activity in response latency during a target-tracking task using a commander. The experimental results showed that the prefrontal cortex was activated by the cognitive stress associated with latency and that the premotor cortex was deactivated when visual and somatosensory information could not be integrated due to latency.