The Effect of Multisensory Pseudo-Haptic Feedback on Perception of Virtual Weight

Providing realistic haptic feedback of virtual objects is critical for immersive VR experience, and there have been many approaches to simulate haptic properties. Most of them, however, are limited to a narrow modulation range of simulated perception. To overcome this limitation, the current paper examines the effect of multisensory pseudo-haptic feedback that combines control-to-display (C/D) ratio manipulation and electrical muscle stimulation (EMS) on simulated weight perception. In two experiments, we independently manipulated the C/D ratio and EMS status and observed the effects on the absolute and difference thresholds of simulated weight perception. From the absolute thresholds results, we specify the effective range of C/D ratio that can successfully induce weight perception and show that the range can be more than twice widened by multisensory pseudo-haptic feedback. Furthermore, we demonstrate that the sensitivity to weight difference increases as the standard C/D ratio decreases from the difference thresholds results, which provides practical design guidelines for assigning multiple levels of weight to virtual objects. This study contributes to understanding the psychological effects of multisensory pseudo-haptic feedback on simulated weight perception in virtual reality.


I. INTRODUCTION
The development of light-weighted and portable [31] virtual reality (VR) head-mounted displays (HMDs) has increased accessibility and usability of VR for the general public [2], and drawn researchers' attention. Furthermore, since the COVID-19 pandemic, the usage of VR in everyday life has increased tremendously, including virtual conferences and meetings [42]. Based on this development and social needs, VR applications that provide entertainment [22], [27] and productivity (e.g., vSpatial, Oculus Medium) are being developed actively. Although VR has great potential in providing new experience and opportunities, there are still many issues left to be solved for VR to be applied in various The associate editor coordinating the review of this manuscript and approving it for publication was Shovan Barma . fields such as industry, health-care system, teleoperation, and remote collaboration [35], [53]. Foremost of the issues is that VR needs to provide more immersive and realistic haptic experiences to augment the sense of real-world in the virtual environment [39], and to let users acquire more information and react properly as in the real-world. For this purpose, companies and researchers are taking various approaches to adopt realistic haptic experience in the virtual environment [6], [23].
A sense of weight is one of the most important haptic properties that contribute to realistic interactions with virtual objects and immersive experience [8], [29]. Most of the methods that augment the sense of weight on virtual objects are based on manipulating either somatosensory or visual information. For instance, electrical muscle stimulation (EMS) manipulates somatosensory information by VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ directly stimulating the muscle areas that are used when lifting an object [17], [29], [30]. For visual manipulations, the Control-to-Display (C/D) ratio is often changed to provide pseudo-haptic feedback on virtual artifacts [23], [36]; Studies have demonstrated that applying a visual offset between a virtual hand and the real hand while lifting a virtual object simulated the sense of weight [39], [44]. Although these methods have been successful in changing participants' weight perception of virtual objects, the size of modulation was limited to only about ±5 g [44], which is rather small for people to detect the differences in weight. Due to this limitation, it has been hard to assign multiple levels of weight on virtual objects by manipulating information in a single sensory domain. Therefore, in the current study, we propose multisensory pseudo-haptic feedback that combines both visual and somatosensory manipulations and precisely measure the effect of the feedback on simulated weight perception. Our approach simulated weight perception by visually manipulating the C/D ratio of hand movements holding a virtual object. In addition, we adopted the EMS to directly change the somatosensory information for a wider modulation range of simulated weight perception. In a series of experiments, we manipulated the C/D ratio and EMS independently and observed the effects on the absolute and difference thresholds of simulated weight perception. The absolute threshold of weight indicates the minimum level of stimulus that people can barely perceive the presence of weight, whereas the difference threshold of weight indicates the minimum difference in stimulus level between two objects that people can barely detect the weight difference [26]. In the absolute threshold experiment, we aimed to investigate the effects of the C/D ratio and EMS on simulated weight perception of virtual objects and verify whether and how multisensory pseudo-haptic feedback widens the modulation range of simulated weight perception. In the difference threshold experiment, we measured the minimum difference in C/D ratio between two objects that people can detect the difference, and examined how the difference threshold changed across standard C/D ratio values. Based on the results of two experiments, we demonstrate the psychological effect of multisensory pseudo-haptic feedback on detection of weight as well as sensitivity to weight changes, and provide design guidelines for simulating multiple levels of weight experience in VR.
We propose that our research makes the following contributions on generating simulated weight perception in VR with pseudo-haptic feedback: • By using multisensory pseudo-haptic feedback that includes both visual and somatosensory stimuli, we specify the effective range of modulation for each type of feedback, as well as the widened range of modulation when both types of feedback are combined.
• By measuring both the absolute and difference thresholds of weight, we provide detailed information about how multisensory pseudo-haptic feedback influences detection of weight per se, as well as the sensitivity to changes in weight.
• Based on the results of two experiments, we show a possibility of augmenting multiple levels of weight experience and provide design guidelines for future VR application.

II. RELATED WORK
Since VR has become the emerging technology nowadays, methods of assigning a sense of real-world experience (e.g., weight, stiffness, texture, air-flow) on virtual objects have been actively studied [14], [36], [41]. Pseudo-haptic feedback, which simulates haptic sensations by incorporating feedback in other sensory modalities, has been studied and adopted in various applications [6], [15], [18], [23]. For instance, visual manipulations for simulating friction and stiffness features were widely used in game contexts when the character passes obstruction or in 2D interfaces to provide feedback of users' command [23]. Among many pseudo-haptic feedback methods, visual and somatosensory manipulations have been most often used to induce haptic illusions of users [3], [24], [28]. Since the current study investigates the effect of multisensory pseudo-haptic feedback that merges both visual and somatosensory manipulations, related work will also be reviewed in three sections: visual, somatosensory, and multisensory pseudo-haptic feedback.

A. VISUAL PSEUDO-HAPTIC FEEDBACK
Humans get information about the position of the body from vision, but also from proprioception, the ability to perceive self-movement and body position without seeing it [19], [51]. The mismatch between information from vision and proprioception is commonly used to design VR pseudo-haptic feedback [48]. Since VR HMDs can provide a visual space completely different from the real world, many VR pseudo-haptic feedback studies used the location offset between the real hand and the virtual hand to induce a mismatch between vision and proprioception, which is perceived as a weight perceptual illusion [18], [39], [40], [44]. This visual pseudo-haptic feedback is advantageous, since it is easily implemented with graphical programming without requiring additional components on the controller to simulate haptic sensations. For instance, Rietzler et al. developed a visual pseudo-haptic feedback model that augments weight by using height offset between the real-hand and the virtual hand [39]. Through a bowling game with the pseudo-haptic feedback method applied on the balls, they found that participants experienced a higher level of presence and immersion in the VR content when the pseudo-haptic feedback was applied. However, previous visual pseudo-haptic feedback methods that manipulated the C/D ratio are limited in that they require user's visual attention on the target object, and the range of modulation effect is rather narrow [39], [44]. Samad et al. showed that the visual pseudo-haptic feedback could modulate the reference weight (i.e., the weight of the 5130 VOLUME 10,2022 VR controller) about ±5 g, which is very small for people to discriminate the weight difference. This might be partly due to the fact that the pseudo-haptic feedback based on the mismatch between visual information and proprioception is only effective in the near-body area [48].

B. SOMATOSENSORY PSEUDO-HAPTIC FEEDBACK
Unlike software-based methods reviewed in the previous section, hardware-based methods directly change the somatosensory information of the body area that needs to perceive the target sensation [5], [20], [29], [49]. Simulating the sense of weight or force feedback has been often studied by using hardware-based methods. For instance, Grabity provided a sense of weight and grasping by stimulating mechanoreceptors in fingers with a wearable haptic device [5]. Also, the forward and backward movement of a motor in the cylindrical rod attached to the controller was used to provide dynamic sense [54]. Other methods used wind resistance to provide pseudo-haptic feedback by attaching propellers to the controller [16], [55].
Another important method to simulate the sense of weight on virtual objects is using EMS to directly stimulate the muscles of the user [17], [29], [30]. EMS is a device that artificially induces muscle contractions through electrical signals. Lopes et al. have conducted multiple studies using EMS to simulate a sense of presence and weight on virtual artifacts in VR. They attached the EMS pads on various areas of the arm, with each area being assigned for a specific sense. For instance, the EMS pad located on the shoulder was to simulate the sense of friction, whereas that on the triceps/biceps was to simulate the light or heavy weight of a virtual object. This approach has augmented not only the sense of weight or force feedback but also the presence of virtual objects even when the EMS was not applied.
The results of many previous studies indicated that somatosensory pseudo-haptic feedback successfully generated a target sensation and provided an immersive, compelling experience to users [5], [29]. However, the methods that attach additional modules on the controller or body areas could be problematic for long-term usage due to heavy weight [25] or discomfort. Also, each of these modules has a specified target sensation which is limited to specific situations with less generalizability.

C. MULTISENSORY PSEUDO-HAPTIC FEEDBACK
Pseudo-haptic feedback using multisensory stimulation was investigated in many studies [12], [33], [34], [52]. These studies have shown that multisensory pseudo-haptic feedback generated the experience of targeted sensation more immersive and realistic than feedback in a single sensory modality [6]. Visual stimuli are most often used with tactile stimuli to provide multisensory pseudo-haptic feedback [12], [34]. For instance, Pezent et al. developed a Tasbi, a haptic wearable device that generates vibration and squeezes stimuli to provide haptic feedback of visual events in virtual reality. Auditory stimuli are sometimes used FIGURE 1. An illustration of our multisensory pseudo-haptic feedback that combines visual pseudo-haptic feedback that manipulates the C/D ratio with somatosensory pseudo-haptic feedback that stimulates the arm muscle with EMS. From the left to right panels, it shows the sequential movement of arm lifting.
with visual stimuli to provide multisensory pseudo-haptic feedback [33], [52]. Park and Kim experimented with a virtual object that had a rough surface and provided a congruent or incongruent auditory stimuli when the object passed the participants' finger. They used a questionnaire to evaluate their pseudo-haptic feedback and found that the multisensory feedback with visual and auditory stimuli provided a stronger sensation of roughness.
Even though these studies showed that multisensory pseudo-haptic feedback can generate a stronger simulated sensation [38], they did not provide detailed information on how to modulate these effects, which is crucial for pseudo-haptic feedback to be actually used in real-world VR applications. Also, multisensory pseudo-haptic feedback to simulate weight perception of virtual objects has not been studied, to the best of our knowledge. Therefore, the current study focuses on investigating the modulation range and sensitivity of simulated weight perception with multisensory pseudo-haptic feedback.

III. DESIGN OF MULTISENSORY PSEUDO-HAPTIC FEEDBACK
Humans perceive the weight of an object through somatosensory and visual information [56]. The visual information regarding weight consists of form (e.g., shape, size, and color) and motion (e.g., velocity, acceleration, and displacement) factors [13], [21], [32], [43]. In this paper, we focused on the velocity and displacement of objects, which are motion factors, to make users perceive the weight of a virtual object. We also used EMS to change somatosensory information and observed if it could widen the modulation range of the simulated weight perception. The specific design and implementation of multisensory pseudo-haptic feedback are described as follows.

A. VISUAL PSEUDO-HAPTIC FEEDBACK DESIGN
Visual pseudo-haptic feedback has often been described as an illusion or a metaphor of the real-world sensation [9], [39], [41], [44]. Many studies have used the mismatch between the movement or position of the virtual hand and the real hand to induce the illusion of weight perception [35], [40], [41], [48]. Based on these studies, in the current study, we used linear interpolation on the hand position of each VOLUME 10, 2022 frame to generate the temporal mismatch between visual information and proprioception to simulate the sensation of heavy weight felt by lifting a real object.
The visual offset effect was implemented based on the linear interpolation method (equation 1), which calculates the interpolated position (x, y) from the known two 2D coordinates, (x 1 , y 1 ) and (x 2 , y 2 ). To adapt this equation in the 3D virtual environment, we altered it as in equation 2. In equation 2, 'p' denotes the position values computed in 3D coordinates (x, y, z). The position value for the virtual hand to be presented in the current time (p virtual t ) is calculated by applying the C/D ratio at the distance between the position of the virtual hand in the previous time (p virtual t−1 ) and the currently tracked position of the real hand (p real t ). This operation was performed for each delta time (frame per second in Unity) and the virtual hand gradually reached the real hand position with a variable time delay. As the C/D ratio gets closer to zero, the virtual hand needs more time to reach the real hand position and makes the movement of the hand and the cube visually slower than the real motion. (2) In this study, different levels of C/D ratio between zero to one was applied to the virtual hand and the cube to make the visual display of the hand movement slower than reality (see Figure 1). Since our visual pseudo-haptic feedback simulated weight perception via the difference in velocity between the real and the virtual hand in vertical lifting movements, rotation of the hand was not considered in the current study. Also, we applied the visual offset effect only when the participant was grabbing and manipulating the cube with the right hand. We excluded the left virtual hand and did not apply the visual offset effect on the cube falling event, to minimize the effects of other confounding variables. Thus, if participants released the cube in the air, the cube appeared in the initial (center) position without falling.
In most previous studies, the C/D ratio was applied to the final height difference (offset) between the virtual and real hand after lifting an object [39], [44]. However, in the current study, we applied the C/D ratio to the velocity of virtual hand movement by contracting distance in each frame. Since the C/D ratio is adapted in each frame in our visual pseudo-haptic feedback, even though the values might seem a bit lower than those used in previous studies, the actual outcome of offset is not as extreme. To verify that our implementation made users perceive weight rather than an error, and to probe the appropriate C/D ratio range, we first conducted an exploratory experiment before the main experiments.

B. SOMATOSENSORY PSEUDO-HAPTIC FEEDBACK DESIGN
EMS was used on both biceps and triceps muscles to simulate the weight of a virtual object in previous studies [17], [29], [30]. In our study, we expected users to perceive heavier weight when they lift a cube with a visual offset effect. Therefore, we needed to generate muscle contraction on users' triceps and relaxation on biceps, which would make their arm go down and feel harder to lift a virtual object. We attached the EMS pad on the triceps of participants to generate this effect (see Figure 1) [29], [30]. For implementation, we used the TENS 7000 device (frequency: 80Hz, pulse width: 50µs) for EMS signal generation and Raspberry pi 3 B+ to control the relay module. Specifically, we developed the EMS On/Off function to be synchronized with the object grab event in VR by communicating with the Unity application through the REST API.
In order to determine the appropriate electric current level to be used for EMS On condition for each participant, there was a calibration procedure prior to the experiment. We attached EMS pad and asked participants to report their subjective experience to changes in the electric current level (e.g., painful, weird, stimulated), such that the appropriate current level to be used in the experiment could be set for each participant. The average current level setting used in experiments was 17 mA (range: 10-30 mA). During the experiment, a static current level calibrated for each participant was applied in EMS On condition to prevent potential safety or discomfort issues.

C. EXPLORATORY EXPERIMENT
The exploratory experiment was conducted to investigate the influence of different levels of C/D ratio and EMS status on users' subjective experience, and to set the appropriate range of parameters for subsequent experiments. For instance, an extremely low level of C/D ratio might be perceived as an error rather than weight, or EMS might be particularly distracting or uncomfortable to users when combined with a certain range of C/D ratio. Therefore, we first conducted an exploratory experiment in which a combination of a wide range of C/D ratio values (i.e., 0.005, 0.05, 0.075, 0.09, 0.1, 0.15, 0.2, 0.25, 0.3, 0.4, 0.5, 0.6 and 1) and the EMS status (on/off) was presented in a random order (two repetitions) and participants' weight perception response was recorded in each trial (see Figure 2). In addition, their subjective experience of sensory, realism, distraction and control factors from Witmer-Singer presence questionnaire (7 point-Likert scale) was asked in each trial [50]. The questions were answered inside the VR environment to keep the sense of presence and immersion on the task. The questions used in the experiment are as follows.
1) How compelling was your sense of objects moving through space? (Sensory Factor) 2) How much did the simulated weight on the virtual object seem consistent with your real-world experiences? (Realism Factor) 3) How distracting was the control mechanism? (Distraction Factor) 4) How much delay did you experience between your actions and the expected outcomes? (Control Factor) In order to implement a virtual environment and visual pseudo-haptic feedback, we used the Oculus Rift S (2560 × 1440 resolution, 115-degree field-of-view, 80Hz refresh rate) for HMD and Unity 3D engine for VR programming. In the virtual environment, a yellow cube was placed on the center of a table in a virtual environment and participants manipulated the virtual right hand with a controller to grab and move the cube up and down (see Figure 1 and 2), experiencing and evaluating the weight of the cube. The Oculus Rift S controller held in the participant's hand was used for tracking the position of the real hand. Participants were instructed to lift the cube up to the shoulder height and experience it at least for three seconds, and respond whether they felt the weight of the virtual cube or not, as compared to the default weight of the controller, by placing the cube on the response area (('Yes') or ('No') area).
A total of 20 participants (11 male; mean age 26.5 (SD = 2.32); one left-handed; four had no experience in VR; none of them had experience in EMS) who did not have any potential problems in using VR HMD and EMS was recruited and received a $5 gift card for participation. The results indicated that the visual offset effect within a certain range of lower C/D ratio values successfully provided weight perception of a virtual object (see Figure 3A). Participants responded that they perceived the weight (over 50 % of probability) of a virtual object when the C/D ratio was under 0.2. EMS also had a significant effect on simulated weight perception, increasing the probability of weight perception in intermediate C/D ratio values. On the other hand, we also observed the negative effects of EMS and C/D ratio manipulations on users' subjective experience. Extremely low levels of C/D ratio, such as 0.005, significantly disturbed the quality of VR experience by making participants feel less compelling, more distracting, and harder to control (see Figure 3B). Also, combining EMS with visual pseudo-haptic feedback made users feel more distracted, especially when the C/D ratio was higher than 0.2. Thus, it is important to combine the appropriate range of C/D ratio values and EMS to increase the quality of simulated weight experience in VR. VOLUME 10, 2022

IV. EXPERIMENT 1: ABSOLUTE THRESHOLD OF WEIGHT
We conducted the first main experiment to measure the absolute threshold of weight in each condition and verify the hypothesis that our multisensory pseudo-haptic feedback that combines a visual offset effect with EMS would create a wider modulation range of simulated weight perception. Even though we already obtained the results in the exploratory experiment that supported this hypothesis, it was limited by a small number of trials (two repetitions) for each condition. Thus, in Experiment 1, we focused on the smaller set of C/D ratio values and increased the number of trials to 20 for each condition, in order to obtain more accurate estimates of the absolute threshold of simulated weight perception. Based on the results in the exploratory experiment, we excluded extremely low C/D ratio values that disturbed the quality of VR experience and sampled more frequently within the range that was effective in inducing weight perception, which resulted in seven C/D ratio values (0.05, 0.1, 0.15, 0.2, 0.4, 0.7, 1) to be used in Experiment 1. By using the method of constant stimuli [46], we measured the minimum level of C/D ratio manipulation that can be detected as simulated weight, separately for EMS On and Off conditions. After the weight perception task, a survey and a short interview about participants' experience of pseudo-haptic feedback were conducted.

A. PARTICIPANTS
A total of 21 healthy adults (10 male; mean age 24.47 (SD = 2.92); two left-handed; four had no experience in VR; five had experience in EMS) recruited from a university participated in the experiment and received a $15 gift card. All of the study protocols and methods were approved by the university's Institutional Review Board (IRB), and each participant provided written informed consent.

B. PROCEDURE
The overall structure and procedure of Experiment 1 was similar to that of the weight perception task in the exploratory experiment (see Figure 2), with the following exceptions. First, before starting a task and after every 40 trials, a test phase was inserted in which participants manipulated a blue virtual cube without any pseudo-haptic feedback to get used to and to be reminded of the default weight of the controller (baseline condition). Second, participants were asked to manipulate the virtual cube at least for three seconds before responding whether they perceived the weight of the cube as compared to the baseline condition or not, in order to provide enough time for more accurate evaluation of the experience. Third, the number of trials in each condition was increased to 20 to get more accurate estimates. Thus, a total of 280 trials were presented in a random order, consisting of seven levels of C/D ratio (0.05, 0.1, 0.15, 0.2, 0.4, 0.7, 1) and two types of EMS status (On, Off), repeated 20 times each. After the weight perception task, a survey based on the Witmer-Singer presence questionnaire [36], [37], [50] was conducted. In the survey, we asked four questions in 7-point Likert scale about comfortableness and disembodiment of their overall experience of EMS and visual pseudohaptic feedback. Also, a short interview followed in which participants verbally described their experience in more detail. The questions used in the survey are as follows.
1) When the (Visual/Multisensory) stimulus was applied, it felt like my real hand and my hand in virtual reality were separated. (Disembodiment Factor) 2) When the (Visual/Multisensory) stimulus was applied, the experience of lifting an object was comfortable.
(Comfortable Factor) C. RESULTS

1) WEIGHT PERCEPTION TASK
We used two-way repeated-measures ANOVA to analyze the effects of C/D ratio and EMS on weight perception responses (see Figure 4). The results showed that there was a significant main effect of C/D ratio on weight perception responses (F(2.06, 41.19) = 54.93, p < .001, η 2 p = .733, Greenhouse-Geisser corrected), with the probability of simulated weight perception increasing as the C/D ratio decreased. Also, the main effect of EMS was significant (F(1, 20) = 10.92, p = .004, η 2 p = .353, Greenhouse-Geisser corrected), with a higher probability of simulated weight perception when EMS was enabled (60.1 %) than when it was disabled (40.4 %). There was also an interaction between C/D ratio and EMS status (F(2.67, 53.4) = 4.27, p = .011, η 2 p = .176, Greenhouse-Geisser corrected). The pairwise comparisons indicated that combining EMS with visual pseudo-haptic feedback significantly increased the probability of simulated weight perception at the C/D ratio range between 0.1 and 1 (t(20)s > 2.15, ps < .05), but not at the lowest C/D ratio (0.05) (t(20) = 1.29, p = .213). All of these results are consistent with those obtained in the exploratory experiment, and confirm that the visual offset effect within a certain range of lower C/D ratio values can successfully simulate the perception of weight, and multisensory pseudo-haptic feedback that combines EMS with the visual offset effect significantly increases the probability of weight perception.
To further examine how much the multisensory pseudohaptic feedback widened the modulation range of simulated weight perception, we calculated and compared the absolute thresholds of simulated weight perception in each EMS condition (On, Off). First, a psychometric function was fitted to the averaged weight perception responses in each condition, then the C/D ratio value that corresponds to the 50 % probability of weight perception on the fitted curve was set as the absolute threshold of that condition. The result showed that participants' absolute threshold of simulated weight perception was significantly higher (t(20) = 2.57, p = .018) when EMS was presented with the visual offset effect (C/D ratio = 0.46) as compared to when the visual offset effect was presented alone (C/D ratio = 0.18). This suggests that our multisensory pseudo-haptic feedback that  combined visual and somatosensory stimulation widened the modulation range of simulated weight by about 2.5 times.

2) SURVEY
For survey results, we used Wilcoxon signed-rank tests to compare the comfortableness and disembodiment scores for single and multisensory pseudo-haptic experience. The results indicated that participants felt that the visual offset effect only condition (M = 4.76, SD = 1.6) was significantly more comfortable than when EMS was applied together (M = 3.85, SD = 1.42) (z = −2.247, p = .025). In terms of disembodiment, on the other hand, participants' ratings were not different for visual (M = 3.38, SD = 1.39) and multisensory pseudo-haptic feedback (M = 3.76, SD = 1.6) (z = −1.215, p = .224).

3) INTERVIEW
Each participant's interview session was recorded and transcribed with anonymization. Two coders independently categorized the transcribed comments into pre-selected four main categories (visual-positive, visual-negative, EMS-positive, EMS-negative), based on whether the comment was about the visual effect or EMS, and whether the description was positive or negative. Then, the two coders discussed and assigned each comment to subcategories as in the Table 1. The table shows the four main categories and their subcategories with the representative comments and the number of participants who mentioned it. Many participants responded that the visual effect was consistent with the weight experience in real-world. Also, many mentioned that EMS was especially helpful for simulated weight perception when the visual effect was ambiguous. However, there were also negative comments that the visual effect caused disembodiment when the offset between virtual and real hand was too big, and that EMS felt uncomfortable or unfamiliar.

D. DISCUSSION
The goal of Experiment 1 was to specify the effective range of C/D ratio values by calculating the absolute threshold in each condition, and to verify the hypothesis that our multisensory pseudo-haptic feedback that combines visual and somatosensory stimulation would create a wider modulation range of simulated weight perception. Consistent with the results from the previous studies, our results indicated that although visual pseudo-haptic feedback alone can simulate weight perception on a virtual object, it is limited to a narrow range of modulation; the absolute threshold of simulated weight perception was 0.18 when visual pseudo-haptic feedback was presented alone. When EMS was presented with visual pseudo-haptic feedback, on the other hand, the absolute threshold of simulated weight perception increased to 0.46. This supports our hypothesis and provides strong evidence that adding EMS on top of C/D ratio manipulations can widen the modulation range of simulated weight perception by 2.5 times.
The survey results indicated that visual pseudo-haptic feedback provided more comfortable experience than somatosensory stimulation (EMS), with no significant difference in experience of disembodiment between two methods. We also obtained more detailed comments and insights from the interview. Many participants agreed that the sense generated by the visual pseudo-haptic feedback matched the sense of real-world weight, and EMS seemed to further support the simulated weight perception. At the same time, there were several negative comments about visual and somatosensory stimulation, especially regarding uncomfortable or unfamiliar experience induced by EMS. These results suggest that it VOLUME 10, 2022 would be good for the quality of simulated weight experience to manipulate C/D ratio levels for the main part, while occasionally adding EMS to supplement and magnify the effect.

V. EXPERIMENT 2: DIFFERENCE THRESHOLD OF WEIGHT
In Experiment 1, we measured the absolute threshold of weight to elucidate the range of C/D ratios that can successfully induce the perception of weight of a virtual object. We found that the multisensory pseudo-haptic feedback including both visual offset and EMS can more than double the range of modulation. However, absolute thresholds do not provide any information about how sensitive users are to small differences in weight simulated by pseudo-haptic feedback. In the context of simulated weight perception, the difference threshold, or just noticeable difference (JND), is defined as the smallest change in C/D ratio that users can detect as difference in weight. Specifying difference threshold values and understanding how the difference threshold changes across standard C/D ratio values is critical for designing multiple levels of weight experience in VR application. Thus, in Experiment 2, we let participants compare the weight of two virtual objects, and measured the difference threshold of weight perception across different levels of standard C/D ratio values (0.05, 0.1, 0.15, 0.2, 0.4, 0.7, 1) and EMS status (On, Off).
As in Figure 5, two yellow cubes were placed on either side of a table with one response ('Heavy') area on the center. Participants were asked to lift each cube up and down and compare the weight of them, and respond by placing the heavier cube on the ('Heavy') area, or leaving the cubes as they were if both cubes felt the same in terms of weight.
In order to measure the difference threshold of weight, we used the maximum likelihood procedure (MLP), which is one of the adaptive methods commonly used to measure the JND [10], [26], [45]. In the MLP, the difference threshold is estimated for each standard C/D ratio in a block of trials. Within each block, the same specific standard C/D ratio is applied to one of the cubes, whereas the C/D ratio applied to another cube changes dynamically across trials. The MLP generates a C/D ratio value to be applied to another cube FIGURE 6. An example of how a C/D ratio generated by the MLP changed across trials within a block, when the standard C/D ratio was 0.15, and EMS was On or Off. Each horizontal dotted line indicates the mean of the C/D ratio values presented within each block. The difference between the mean of the generated C/D ratio values and the standard C/D ratio was calculated to be the difference threshold for the standard C/D ratio. The square marker symbol indicates catch trials in which the lowest level of C/D ratio (0.01) was presented to prevent underestimation of the false alarm rate.
for each trial based on the participant's previous responses within the block, which makes it possible to estimate the difference threshold for the current standard C/D ratio with a smaller number of trials than other adaptive methods, such as staircasing [26].
For parameters of the MLP, we used 0.1 for the slope of a logistic psychometric function that affects the generated C/D ratio level, and set the range of generated C/D ratio level from 0.01 to the current standard C/D ratio [10]. Each MLP block consisted of a total of 18 trials, including four randomly inserted catch trials in which the lowest level of C/D ratio (0.01) was presented to prevent underestimation of the false alarm rate [10], [11]. These parameters were adjusted and selected through a pilot test with five participants. Figure 6 illustrates an example of how a C/D ratio generated by the MLP changed across trials within a block, when the standard C/D ratio was 0.15. The difference between the standard C/D ratio and the mean of the generated C/D ratio values within the MLP block was calculated to be the difference threshold for the standard C/D ratio.

A. PARTICIPANTS
A total of 20 participants (14 male; mean age 26.95 (SD = 2.09); 2 left-handed; 5 had no experience in VR; 4 had experience in EMS) from a university participated in the experiment and received a $10 gift card. All of the study protocols and methods were approved by the university's Institutional Review Board (IRB), and each participant provided written informed consent.

B. PROCEDURE
The initial procedures of task instruction, EMS level calibration, and the test phase were the same as in Experiment 1. In the main experiment, participants were asked to lift each cube up and compare the weight of them, and respond by placing the heavier cube on the center ('Heavy') area, or leaving the cubes as they were if both cubes felt the same in terms of weight. During probing, they were instructed to lift up each cube one at a time, and put the cube down in its original position and lift another cube for comparison. Participants were required to examine the cubes at least for three seconds to proceed to the next trial, in order to prevent premature responses. A total of 252 trials were conducted, consisting of seven levels of standard C/D ratio (0.05, 0.1, 0.15, 0.2, 0.4, 0.7, 1) and two types of EMS status (On, Off), with 18 trials in each block of MLP. The order of MLP blocks was randomized for each participant.

C. RESULTS
The difference threshold data were analyzed with two-way repeated measures ANOVA. The results showed that there was a significant main effect of standard C/D ratio on difference thresholds (F(1.46, 27.74) = 46.671, p < .001, η 2 p = .711, Greenhouse-Geisser corrected), indicating that the sensitivity to weight difference changed across standard C/D ratio values (see Figure 7). Overall, the difference threshold increased as the standard C/D ratio increased. Confirming this, the difference threshold values across standard C/D ratios fit by polynomial regression revealed a significant linear component (F(1, 19) = 68.45, p < .001, η 2 p = .783) and quadratic component (F(1, 19) = 8.829, p = .008, η 2 p = .317). This suggests that as the standard C/D ratio increases, users require bigger changes in C/D ratio to detect the difference in weight. The main effect of EMS (F(1, 19) = 1.52, p = .232, η 2 p = .074, Greenhouse-Geisser corrected) and interaction between C/D ratio and EMS (F(1.94, 36.95) = 1.49, p = .239, η 2 p = .073, Greenhouse-Geisser corrected) were not significant, indicating that EMS status had little influence on the smallest change in C/D ratio that users can detect as difference in weight.

D. DISCUSSION
We conducted Experiment 2 to specify difference thresholds of weight induced by multisensory pseudo-haptic feedback, and to elucidate how the difference threshold changes across standard C/D ratio values. The results showed that the difference threshold changed across standard C/D ratio values, such that the sensitivity to weight difference decreased as the standard C/D ratio increased. For example, when the standard C/D ratio applied to one of the cubes is 0.2, another cube's C/D ratio should be different from 0.2 at least by 0.07, for users to detect the weight difference between them. When the standard C/D ratio increases to 0.4, on the other hand, another cube's C/D ratio should be different from 0.4 at least by 0.15, for users to discriminate the weight difference. This result is consistent with the classic psychophysics theories such as the Weber-Fechner law or the Stevens's power law [7], [47] in that even the strength of physical stimuli (C/D ratio) changed linearly, participants' psychological perception (weight) of it changed non-linearly. In other words, the same amount of change in C/D ratio had a smaller psychological effect on detecting weight difference when the standard C/D ratio increased.
In contrast to the significant effect of EMS on the absolute threshold data in Experiment 1, we observed no significant effect involving EMS in the difference threshold data in Experiment 2. The seemingly inconsistent results, however, can be perfectly explained by looking into the task design of Experiment 2. The purpose of Experiment 2 was to obtain difference threshold of C/D ratio (smallest change in C/D ratio levels), not the difference threshold of EMS (smallest change in electric current levels). Thus, in each MLP block of trials, the EMS status applied to two cubes was always the same, both On or both Off, and only the C/D ratio difference between the two cubes systematically changed across trials. In other words, participants never directly compared the weight of two cubes that had different EMS status (On, Off), or different electric current levels. In order to obtain the difference threshold of EMS, an additional experiment will be needed in which the C/D ratio applied to two cubes is kept constant, and only the difference in electric current levels for the two cubes systematically changes. Taken together, the results of Experiment 2 illustrate how the difference threshold of weight changes across standard C/D ratio values, while the EMS status is controlled within comparisons.

VI. GENERAL DISCUSSION
In this paper, we present multisensory pseudo-haptic feedback that uses both hardware and software methods to induce weight perception of virtual objects. Different levels of a visual offset effect were used for a software method, and EMS that stimulated arm muscles was used for a hardware method. Utilizing the multisensory pseudo-haptic feedback, we conducted two experiments measuring the absolute threshold and difference threshold of simulated weight perception, respectively. It was hypothesized that our multisensory pseudo-haptic feedback that combines visual and somatosensory stimulation would generate a wider modulation range of simulated weight perception (Experiment 1), and users' sensitivity to simulated weight difference would VOLUME 10, 2022 vary across levels of multisensory pseudo-haptic feedback (Experiment 2). From the absolute threshold experiment, we found that both C/D ratio manipulations and EMS are effective in making people perceive the weight of a virtual object: Participants perceived the presence of weight of a virtual object when the C/D ratio value was in a certain lower range (under 0.18), and that range became more than twice wider (under 0.46) when EMS was added on top of the visual offset effect. Moreover, we elucidated that the additional effect of EMS on simulated weight perception is significant only in a certain range (0.1 to 1) of C/D ratio.
In the difference threshold experiment, we measured the smallest change in C/D ratio that users can detect as difference in weight induced by multisensory pseudo-haptic feedback. The results showed that the difference threshold increased as the C/D ratio of the standard stimulus increased. That is, participants were more sensitive to subtle changes in C/D ratio when the standard stimulus had a lower C/D ratio value. This suggests that when designing different levels of weight experience for multiple objects in VR, the differences in C/D ratio between objects should be scaled to the C/D ratio applied to the standard stimulus for users to actually perceive the weight difference.
Utilizing the absolute and difference threshold results obtained in Experiment 1 and 2, multiple levels of simulated weight experience for virtual objects can be designed effectively. For instance, only visual pseudo-haptic feedback with the C/D ratio of 0.05 and 0.1 could be applied to virtual objects A and B, respectively, to simulate different levels of relatively heavy weight. In addition, multisensory pseudo-haptic feedback with EMS and the C/D ratio of 0.2 and 0.4 could be applied to virtual objects C and D, respectively, to simulate different levels of relatively light weight. In this way, various levels of weight could be applied to virtual objects to present useful information for users to perform complex tasks, such as assembling virtual objects or remote collaborations in virtual environment.

A. LIMITATION AND FUTURE WORK
Still, there are several limitations to the current study. One of them is that the EMS status was applied in a binary format (On or Off) with a fixed current level for each participant in our experiments. Variations in the current level of EMS might have generated different effects on the absolute and difference thresholds of weight. However, there are huge individual differences in sensitivity to electrical stimulation, and higher current levels are prone to increase the level of discomfort and distraction as confirmed in our survey and interview results. Thus, we did not include the current level variations in the current study, and focused on the additional effect of EMS at an acceptable current level calibrated for each participant. In future studies, the effect of the combination of different current levels and C/D ratio manipulations on simulated weight perception could be explored.
Along with a fixed EMS current level, we applied linear C/D ratios between zero to one in the current study. Using nonlinear C/D ratios, rather than linear C/D ratios, might result in different threshold values for simulated weight perception and therefore might contribute to providing a wider C/D ratio range for weight augmentation. Also, C/D ratios larger than one would make the virtual hand move faster than the real hand, which might make people perceive lighter weight. Since we focused on the effect of multisensory pseudo-haptic feedback on simulated weight perception in this study, we could not examine above variations in order to accommodate all necessary conditions within the limited experiment time. Testing the effect of nonlinear C/D ratios or C/D ratios larger than one on simulated weight perception would be an interesting and important topic for future studies. Replicating the pattern of results obtained in the current study with these novel C/D ratio manipulations with a larger sample size would greatly strengthen our conclusion.
Another limitation is that our experiments were conducted with a single weight condition of the hand-held object; Participants were holding a controller (129 g) in all trials. Holding a differently-weighted real object or holding no real object might affect the absolute and difference thresholds of simulated weight perception, but this manipulation was excluded in the current study to focus on the effects of multisensory pseudo-haptic feedback. This would be an important issue for future VR application, since the advances in the hand-tracking technology enable users interact with their bare hands [1], [4], [44]. In the future study, therefore, we are planning to investigate how the methods of interaction in VR affect the weight perception thresholds and the quality of experience (i.e., immersion and realism).

B. CONCLUSION
In conclusion, the current study provides detailed information about how multisensory pseudo-haptic feedback that combines C/D ratio manipulation and EMS affects user's perception of simulated weight in VR. We first specify the range of C/D ratio that can effectively induce perception of virtual weight, and demonstrate that the range can be more than twice widened by adding EMS on top of the visual offset effect. Moreover, it is shown for the first time in this study that sensitivity to virtual weight difference changes according to the standard C/D ratio, such that users are more sensitive to small changes in C/D ratio when the standard object has a lower C/D ratio value. These results provide practical as well as theoretical implications for designing multiple levels of weight experience in VR.

ACKNOWLEDGMENT
The data and materials for the experiment are available upon request. The authors declare no competing financial interests.