Can Facial Expressions Induce Haptic Perception?

A key challenge in haptics is designing human–human communications involving touch to facilitate positive effects on social interactions. An important consideration in designing social touch is understanding the effect of social stimuli on perception, in addition to that of a physical stimulus, because social touch always involves a partner. This study presents an experiment to demonstrate that facial expressions induce haptic perception. We developed a human–agent interaction system on a display in which participants moved the mouse cursor to click the target icon while the agent behaved as if it pulled the cursor back in the opposite direction, showing either a negative or neutral face. The perceived force during the interaction was quantified by the control display ratio using a psychophysical approach. The results show that the negative face induced a significantly greater perceived force than the neutral face. In addition, the perceived force correlated with the individual's evaluation of the facial expression; that is, the more unpleasant or aroused they perceived the facial expression to be, the more force they perceived. This study sheds light on the design of social touch performed by people who have physical or mediated contact with each other in physical space or cyberspace.


I. INTRODUCTION
A KEY challenge in haptics is designing human-human communication involving touch to facilitate positive effects on social interactions [1], [2].Advances in technology have enabled the augmentation of physical interpersonal touch [3] and touch with a distant partner [4], [5].These technologies have attracted attention under touch deprivation conditions during the coronavirus pandemic [6].
An important consideration for designing social touch is to understand the effect of social stimuli on perception, in addition to that of the physical stimulus, because social touch always involves a partner.Here, we consider that a partner is not only another human but also an artificial agent, such as computer graphics or a robot.Facial expressions are social stimuli that convey emotional states to observers [7], [8].For example, an Fig. 1.Illustration of our human-agent interaction system to study whether facial expressions affect haptic perception.We hypothesize that participants perceive more force from the agent pulling the mouse cursor with a negative face (a) than that with a neutral face (b).
observer may see a person trying to pick up an object.If the face of the person becomes strained, the observer may assume that the object is heavy and offer to help the person.Furthermore, these perceptions and behaviour may be related to individual characteristics.For example, if the observer sees less emotion on the face, they may estimate that the object is lighter.
In this study, we demonstrate that social stimuli induce haptic perception by evaluating the force perceived by participants during interaction with an artificial agent with facial expressions.Fig. 1 illustrates the human-agent interaction system used in this study.The display has a cursor controlled by a mouse and an agent connected by a string.When the cursor moves to the right, the agent pulls the string in the opposite direction.We expected participants to perceive more force when the facial expression of the agent was negative (Fig. 1(a)) than when it was neutral (Fig. 1(b)).We present an experiment to quantify the perceived force using a psychophysical approach and analyze the relation between the perceived force and individual characteristics.

A. Visuo-Haptic Interactions
Presenting sensory stimuli induces haptic sensations, even without physical stimulation of the skin or muscles.Pseudohaptic feedback, a technique for simulating haptic sensations in virtual environments using visual feedback based on user input, has been studied in various contexts [9], [10].Lècuyer et al. demonstrated that the compliance of a virtual spring can be simulated by a technique that distorts visual displacement [11].The technique modulates the control display ratio (CD ratio), defined as the output gain with regard to the input, such as the cursor displacement on the monitor with regard to the mouse displacement on the desk.For example, a user feels as if the mouse descends to a hole with a large CD ratio, while feeling as if the mouse climbed the bump with a small CD ratio [12].The perception resulting from the modulated CD ratio relies on the context.Dominjon et al. reported that a lifted virtual ball felt heavier when the CD ratio was small [13].
According to a review by Ujikoto and Ban from 2021 [10], pseudo-haptic feedback can be categorized into four types of visual stimuli (displacement, surface deformation, color, and size) based on three types of user inputs (displacement, force, and pressing duration).In the proposed system illustrated in Fig. 1, the input falls into displacement, whereas the visual stimulus does not fall into any category defined above.In addition, the researchers have indicated that studies in which pseudo-haptic feedback is used for human-to-human touch experience or social touch are lacking [10].

B. Emotional Expressions in Human-Agent Interactions
Facial expressions, a form of nonverbal communication, can be social signals even if they are unnatural, such as those of a robot or computer graphics [14].They are often used to facilitate social interactions with humans in field and laboratory settings.Most faces of these artificial agents are designed according to the Facial Action Coding System (FACS) built by Ekman et al. [15].It anatomically classifies facial movements according to their appearance and deconstructs them into specific action units.
Hadel et al. designed the 18 facial expressions for a drone consisting of three intensity levels (low, medium, and high) for six emotions (joy, sadness, fear, anger, surprise, and disgust) derived from Plutchik's theory [16] based on FACS to enrich human-drone interactions [17].They asked participants to classify the designed facial expressions into six emotions.The result shows that participants accurately recalled five emotions, while disgust was confused with sadness.In Section III-B1, we present a design of facial expressions by referring to these studies and the determination of the neutral and negative faces in Fig. 1 according to Russell's circumplex model [18], which describes emotion in a two-dimensional (arousal and valence) circular space.
Facial expressions of artificial agents affect human behavior and perception.Bruce et al. reported that a social robot with a computer graphic face made more passersby stop to interact than that without a face [19].Letite et al. demonstrated that a robot with an emotional face made players feel that playing games was more joyful [20].
In the emerging field of affective haptics, researchers have sought to understand human emotions that involve touch interactions.Etzi   subjective measures during exposure to emotional pictures and gentle strokes on the forearm with various textures, including skin-to-skin contact [21].The results revealed that hedonic and sensory haptic ratings were modulated by pictures with both positive and negative valence as well as by neutral pictures.Harjunen et al. investigated the effect of facial expressions of an agent on haptic perception and emotions [22].For example, touch with high-arousal (happiness, anger) expressions increased intensity.Although these studies showed that social stimuli affect haptic perception and emotions, participants did not actively interact with the systems like our setup.Ahmed et al. studied whether the facial expressions of an agent affect touch behavior by measuring the pressure applied to the interface by participants [23].The results show that the duration and intensity of the pressure increased when the agent exhibited facial expressions.
These results implicitly support the validity of our approach for demonstrating socially induced haptic perceptions.As a novel contribution, this study aimed to elucidate the effect of individual characteristics, such as emotional and social sensitivity, on perception.

III. EXPERIMENT
The purpose of this experiment was to demonstrate that the negative face of the agent induces more perceived force than the neutral face in our human-agent interaction system.In addition, we analyze the relation of individual perceived force to the emotions recognized on the faces and Autism Spectrum Quotient (AQ) scores.

A. Design
The system comprised a cursor controlled by the participants and an agent that behaved interactively according to the cursor movement on a display (Fig. 2).Our design concepts for the agent are: 1) to portray the agent not as a participant proxy such as a cursor, but another person; 2) to exert (pseudo) force on the cursor; and 3) to provide social stimuli via facial expressions.We employed the following haptic interaction, mediated with an elastic string, in which the agent was designed such that the agent behaved against the cursor movement.Initially (in the standby state), the agent had the neutral face (Fig. 3(a).As the cursor moved to the right, the agent pulled a string connected to the cursor to the other side.During this pulling motion, the face of the agent changed to a negative face (Fig. 3(b)).We expected participants who were emotionally sensitive to social stimuli to feel a resistance force by observing the face and guessing the emotion of the agent.As a control, the face was not changed during the pulling motion (Fig. 3(c)).We quantified the perceived force using the CD ratio through a psychophysical method (the staircase algorithm).Hereafter, the state in which the face changes to negative is referred to as the dynamic state, whereas the state in which the face remains neutral is referred to as the static state.
To investigate individual characteristics, we measured the emotions recognized from our designed faces and AQ.The recalled emotion was measured by the affect grid [18], which is a self-report scale that measures emotions for a given stimulus as a two-dimensional coordinate (pleasant/unpleasant emotion on the valence axis and aroused/sleepy emotion on the arousal axis).AQ is a self-report scale that measures autistic tendencies; higher scores correspond to higher autistic tendencies [24].
We propose the following three hypotheses for this experiment: H1: The dynamic state induces greater perceived force than the static state.
H2: The perceived force negatively and positively correlates with the recalled valence and arousal, respectively.H3: The perceived force in the dynamic state negatively correlates with the AQ scores.

B. Materials
As shown in Fig. 2, the display (Dell, U2412 M, 1920×1200 px, 60 fps) initially showed a green cursor (15 px in radius), the agent (300×450 px) in the standby state (Fig. 3(a)), and a target blue icon (75×75 px).The cursor and agent were located 720 px from the target icon.A straight brown line (8 px in thickness) representing the string was displayed between the cursor and the agent.The cursor movement was restricted to the lateral direction.The participants were instructed to move the cursor to the icon and click on it using the mouse.After the click, the display returned to its initial state.
When the distance between the cursor and agent, d = A t − C t , exceeded the motion start threshold, d >150 px, the agent started the pulling motion (Fig. 3(b) or (c)).When the distance decreased below the motion stop threshold, d <90 px, the agent stopped the motion and returned to the standby state (Fig. 3(a)).During the pulling motion, the agent followed the cursor in each frame: (1) Here, A t is the position of the agent, C t is the cursor position before the CD ratio processing (invisible on the display), as described in (2), v t is the agent's movement per frame (velocity), and a is an adjustment gain set to 0.01.The values of the motion switch thresholds (d >150 px and d <90 px) and a were determined empirically by the authors so that the pulling motion appeared natural.The CD ratio modulation technique was applied to the cursor movement as follows: where ΔC denotes the relative displacement of the cursor, C t denotes the current cursor position, C t−1 denotes the previous cursor position, and b denotes the CD ratio.The CD ratio b was set to 1 during the standby state, whereas it was set to the variable during the pulling motion (see Section III-B1).

1) Designing Behavior and Facial Expression:
With reference to previous studies [15], [17], we designed 21 facial expressions of the agent using the eyebrows, eyes, nose (wrinkles between the eyebrows), and mouth, based on cartoon-style expressions (see Appendix).The 21 facial expressions consisted of seven types of emotions (six emotions + one emotion) × three levels of intensity (Low, Medium, High).The six emotions were derived from Plutchik's theory [16] (Joy, Sadness, Fear, Anger, Surprise, and Disgust), while the additional emotion was designed by the authors by combining the disgust and angry faces (Anger × Disgust).
The authors evaluated the 21 facial expressions using the affect grid [18] (see the Section III-C2 for more details of the affect grid).As a result, we employed Angry × Disgust with three intensities as they were rated as the most unpleasant and aroused faces.We made an animated face in which the three faces were swapped every 0.25 s and used it in the dynamic state (Fig. 3 B).Regarding the face for the standby and static states, we newly designed a face combining Angry × Low, Sad × Low, and Fear × Low, which were rated as the most neutral or of minimum distance from the origin of the affect grid (Fig. 3(a)  and (c)).
In addition to the facial expressions, we designed the body and motion of the agent to enable participants to understand the movements at a glance (Fig. 3(b) and (c)).During the standby state and pulling motion, two(Fig.3(a)) and three(Fig.3(b) and  (c)) images were swapped every 0.25 s, respectively.See the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
attached videos for our designed behavior and facial expressions of the agent.

C. Measures 1) Perceived Force:
To quantify the perceived force induced by facial expressions, we measured the points of subjective equality (PSE) as a reference in the static and dynamic states using the staircase algorithm.The reference was set to a static state with a constant CD ratio (b = 0.5).
For each trial, participants were exposed to two types of interactions: one was the reference and the other was a probe in which b varied from 0.1 to 2.0 in either static or dynamic states.After the exposure to the interactions, participants were asked, "in which interaction did you feel heavier?" and they answered the first or second (a two-alternative forced choice).Each correct identification of the reference caused b of the probe to increase by the step size, whereas each incorrect response caused b to decrease by the step size.When participants responded incorrectly after a correct response or correctly after an incorrect response, this was termed a reversal, as it caused the direction of the staircase (increasing or decreasing) to reverse.The step size, initially 0.4, was halved at each reversal to a minimum of 0.0125.This continued until eight reversals occurred, and the b value resulting in convergence at PSE was obtained.
We set two types of staircase executions: one was an ascending series that started at a small CD ratio (b = 0.2), whereas the other was a descending series that started at a large CD ratio (b = 1.5) in the probe.In addition, we set the two states for the probe with three repetitions.Thus, 12 executions were performed for each participant.To prevent response bias, a pair of ascending and descending series was executed in parallel, and the order of the states in the probe was randomized.
2) Emotion Recognized From Facial Expressions: To characterize the individual emotional sensitivity to our designed faces, we used the affect grid [18].Participants were asked to evaluate the 22 static faces (21 emotional faces + one neutral face) described in Section III-B1 and seven animated faces.Each animated face consisted of three intensity levels for each emotion, which were swapped every 0.25 s.The order of the faces was randomized.
3) Autism-Spectrum Quotient: To characterize individual sensitivity to social stimuli, we used the AQ test in Japanese [24], [25].It consists of 50 statements, each in a forced-choice format.The statements cover five different subdomains associated with the autism spectrum: Social Skills, Communication Skills, Imagination, Attention to Detail, and Attention Switching.The mean AQ score for the Asperger Syndrome (AS) or high-functioning autism (HFA) group is 37.9, while the mean AQ score for the control group is 18.5, according to the survey conducted in Japan [25].

D. Procedure
Twenty students, 11 men and nine women, ages 22 to 24, belonging to the same institution as the authors, participated in this experiment.The experimental protocol was approved by the Institutional Review Board of the University of Tsukuba (Protocol Number: 2022R620).Before the experiment, the experimenter obtained informed consent after explaining the purpose and procedures of the experiment.Participants were paid an Amazon gift card of 1000 JPY for an approximately 90-min session.
First, participants were asked to complete the training trials along with the experimenter to practice the task.They were then asked to complete test trials consisting of the 12 staircase executions.Next, they were asked to evaluate the emotions recognized on the 22 static faces and the seven animated faces using the affect grid after watching the instruction video about how to use the affect grid.Finally, participants were asked to complete the AQ test and questionnaire.

E. Analysis
The data obtained from the staircase algorithm were the CD ratio b of the PSE in the probe.The mean of the six values of b was computed for each state of each participant.Hereafter, this value is referred to as perceived force index.In this phase, we removed the data whose perceived force index in the static state exceeded the mean ± 3 × standard deviation among the participants as outliers.We also computed the difference in perceived force indexes between the dynamic and static states.Hereafter, this force is referred to as the induced force index.A Wilcoxon signed-rank test (one-tail) was used to evaluate whether the perceived force index in the dynamic state was significantly greater than that in the static state.
The data obtained from the affect grid were two-dimensional coordinates.To obtain the relative emotional sensitivity among the 29 faces, the values of each participant on each axis were transformed into z-scores.Hereafter, these computed values are referred to as recalled valence and arousal.To analyze the correlations of these values with the perceived force index, Spearman's rank correlation coefficient ρ was computed.To analyze the correlation between the AQ score and its five subdomains with the induced force index, Spearman's ρ was computed.
The criterion for significance was set to α = 0.05.We considered that two variables correlate if |ρ| > 0.2; however, the sample size might not have been sufficient to show the significance of weak or moderate correlations, particularly for correlations between the induced force index and the AQ and its subdomain scores.

F. Results
We removed the data of one participant because their perceived force index in the static state exceeded our criterion.Fig. 4 shows the perceived force indexes in static and dynamic states.The results indicate that the perceived force index in the dynamic state was significantly greater than that in the static state (W = 181, Z = 4.0, p <0.0001, r = 1.1, 1 − β = 1.0).
Fig. 5(a), (b), and (c) shows the scatter plots of the perceived force index of each participant corresponding to the recalled valence and arousal, respectively.Table I shows Spearman's ρ indicating that the perceived force index negatively correlates with the recalled valence, while positively correlating with the recalled arousal.The results of the correlation analysis between the perceived force index and recalled emotions supported H2.The correlation coefficients, ρ, were greater than 0.3 (weak correlation).Fig. 5 implies that our face design was reasonable, considering that the face in the dynamic state was evaluated as more unpleasant and aroused than the face in the static state, which corresponds with the evaluation of the authors.One opportunity for further investigation is to study how other types of facial expressions affect the results.For example, it would be valuable to observe how the results are affected by a joyful face, which was evaluated as pleasant and sleepy.As the context is important, the joyful face with a pulling motion might confuse the participants.
The results of the correlation analysis between the induced force index and AQ scores did not support H3.In addition to the lack of significant results, the induced force index did not correlate with the AQ scores, Social Skills, or Attention Switch, Additionally, it positively correlated with Communication Skills and Imagination, while negatively correlating with Attention to Detail.One possible reason is the biased sample for AQ scores.As shown in Fig. 6, the sample does not include values greater than 37.9, which is the mean score of the AS/HFA group reported in [25] whereas there is only one sample with a value greater than 33, which is considered the cutoff point [25].Thus, the current results may only represent a subgroup of the population.
Through observation of the participants and review of the questionnaire, we reacknowledged the importance of context design.To evaluate the effect of facial expressions on haptic perception, we carefully designed the system to provide sufficient context while not providing too much bias.To enrich the context, for example, we could change the target icon to something that the agent does not like, such as a dentist, and provide the agent's preference by inserting a speech balloon saying, "I don't like the dentist."However, this context might prevent us from evaluating the effect of facial expressions.In addition, the design of the agent, including facial expressions and behavior, may also affect the results.It would also be valuable to study the effect of the attractiveness of an agent's appearance on perception.
Additionally, there are opportunities for further investigation.First, this experiment did not consider the effect of input devices on perception.Because no instruction on the input was provided, the participants controlled the mouse as they liked.The faster they moved the mouse and reached the target, the shorter they observed the face of the agent and cursor movement.In addition, the mouse is not a displacement input device but a velocity input device.The gain also relies on the hardware and the operating system.Thus, the time of exposure to the interaction may differ among participants.It is noteworthy that the participant whose data we removed from the analysis controlled the mouse uniquely.The participant quickly moved the cursor to the target, waited for some time, probably observing the agent, and finally clicked.Another limitation is that all the measures in this experiment were subjective.Therefore, it would be valuable to decode the behavior of participants, for example, by introducing a force sensor and an eye tracker to measure the gripping force of the input device and gaze fixation time on the face and analyze their relations with the perceived force.

V. CONCLUSION
The goal of this study was to demonstrate that facial expressions induce haptic perception.Therefore, we presented an experiment with our developed human-agent interaction system on a display in which participants tried to move the mouse cursor to click the target icon while the agent behaved as if it pulled the cursor back in the opposite direction with either a negative or neutral face.The perceived force was quantified using the CD ratio through a psychophysical approach.The results showed that the negative face induced a significantly greater perceived force than the neutral face.In addition, the perceived force correlated with the individual's evaluation of the facial expressions; that is, the more unpleasant or aroused perception of the facial expression, the more perceived force they felt.This study sheds light on the design and engineering of social touch performed by people who have physical or mediated contact with each other in physical space or cyberspace.
We envisage several promising areas for future research.As discussed above, it would be valuable to extend these results to various contexts, facial expressions, and behavior of the agent, and objective and subjective measures to elucidate the key factors that socially affect haptic experience.We also look forward to applying the knowledge developed in this study to estimate individual characteristics such as emotional sensitivity from the quantified perceived force.

Fig. 3 .
Fig. 3. Behavior and facial expressions of the agent: The agent stands with the neutral face in the standby state (a), and the agent with the negative and neutral faces pulls the cursor in the dynamic (b) and static (c) states, respectively.

Fig. 4 .
Fig.4.Perceived force index (CD ratio of PSE for the reference, that is, the static state with the 0.5 CD ratio) in the static and dynamic states.

Fig. 5 .Fig. 6 .
Fig. 5. Relation of the perceived force index to a) the recalled valence and b) recalled arousal.TABLE I SPEARMAN'S ρ BETWEEN THE PERCEIVED FORCE INDEX AND RECALLED EMOTIONS

Fig. 6 (
Fig. 6(a)-(f) shows the scatter plots of the induced force index of each participant corresponding to the AQ scores and the scores of the five subdomains, respectively.Table II shows Spearman's rho indicating that the induced force index is negatively correlated with Attention to Detail and positively correlated with Communication Skills and Imagination.
et al. presented experiments that investigated

TABLE II SPEARMAN
'S ρ BETWEEN THE INDUCED FORCE INDEX AND THE AQ SCORESAND ITS FIVE SUBDOMAIN SCORES 0.5) in the static and dynamic states.A greater PSE indicates greater sensation felt in a state.The median in the static state was 0.5313, which is relatively similar to the reference, while that in the dynamic state was 0.7146 and approximately 1.4 times the reference.The PSE in the dynamic state yielded greater variation than that in the static state, as shown by the interquartile ranges of 0.5490 and 0.7917, respectively.This suggests that the perceived force is affected by individual characteristics, as expected.