Altering Time Perception in Virtual Reality Through Multimodal Visual-Tactile Kappa Effect

The perception of time is highly subjective and intertwined with space perception. In a well-known perceptual illusion, called Kappa effect, the distance between consecutive stimuli is modified to induce time distortions in the perceived inter-stimulus interval that are proportional to the distance between the stimuli. However, to the best of our knowledge, this effect has not been characterized and exploited in virtual reality (VR) within a multisensory elicitation framework. This paper investigates the Kappa effect elicited by concurrent visual-tactile stimuli delivered to the forearm, through a multimodal VR interface. This paper compares the outcomes of an experiment in VR with the results of the same experiment performed in the “physical world”, where a multimodal interface was applied to participants' forearm to deliver controlled visual-tactile stimuli. Our results suggest that a multimodal Kappa effect can be elicited both in VR and in the physical world relying on concurrent visual-tactile stimulation. Moreover, our results confirm the existence of a relation between the ability of participants in discriminating the duration of time intervals and the magnitude of the experienced Kappa effect. These outcomes can be exploited to modulate the subjective perception of time in VR, paving the path toward more personalised human-computer interaction.


I. INTRODUCTION
T HE perception of the temporal properties of the environ- ment is subjective and it is prone to perceptual biases.For example, the perceived duration of a stimulus is affected by internal factors such as cognitive load and arousal [1], [2], by the sensory channel involved in the task [3], by properties of the stimulus such as temporal frequency and speed [4], biological motion [5], [6], and by somatotopic considerations (for instance, somatotopic distance affects simultaneity and temporal interval touch-enabled judgments [7]).At a perceptual level, the spatial distance between two stimuli is known to alter the perception of time.Specifically, the distance in space between two consecutive stimuli induces a time dilation of the inter-stimulus interval (ISI) that grows proportionally with the distance, a phenomenon known as the Kappa effect [8].The Kappa effect was demonstrated in visual, auditory and tactile domains using different experimental protocols [9], [10], [11].However, in the tactile domain, the effect was not clearly identified on contiguous areas of the skin [12].Furthermore, the Kappa effect elicited through multimodal stimulation has received little attention.This is especially true for what concerns concurrent visual-tactile stimulation [13].
In our previous works, we pioneered the investigation of time-space illusions elicited by multimodal visual-tactile stimuli delivered through physical interfaces in the real world.In a preliminary study [13] we compared the Kappa effect in the peripersonal space (PPS) using a wearable device equipped with LEDs and vibrotactile actuators.The study compared unimodal visual (V), tactile (T) and bimodal (VT) stimuli suggesting the presence of the Kappa effect in all modalities.In a second study [14] (currently under review), we developed a method to model the Kappa effect by parameterizing a series of psychometric functions fitted on unimodal V,T and bimodal VT stimuli.
Fig. 1 exemplifies the changes in the psychometric function elicited by the Kappa effect on the evaluation of two complementary ISIs given by 3 consecutive stimuli (the protocol used in our study): with respect to equidistant stimuli (red line), the point of subjective equality (PSE) shifts to greater ISIs ratios (T 1 /T 2 ) when the distance of the first interval is shorter than the second one (blue line) and to smaller ISIs ratios (T 1 /T 2 ) when the distance of the first interval is longer than the second one (green line).
Our recent studies [13], [14] confirmed the presence of the Kappa effect in both the unimodal V and T modalities, showing a larger effect size (i.e.greater temporal distortions) in the visual domain.Interestingly though, the integration of the concurrent temporal information provided by the VT modality did not reduce the illusion.Finally, the study suggested a proportional relationship between a participant's ability to discriminate similar ISI and the magnitude of the Kappa effect measured (i.e. the amount of temporal distortion provided by the difference of the stimuli distance).To the best of our knowledge, there are no research articles that investigated the Kappa Effect elicited through multimodal visual-tactile interfaces in virtual reality (VR).The use of VR is known to elicit the phenomenological sensation of presence.Despite knowing that sensory stimuli are fictitious, a convincing VR system can induce the sensation that it is real, causing users to behave as if they were in a physical environment [15].With simple VR setups, researchers can robustly induce the sensation of out-of-body experiences [16] as well as other body-transfer perceptual illusions such as the elongated-arm illusion and the rubber hand illusion [17], [18].VR thus provides psychologists with an essential tool for the study of the phenomenology of self and self-consciousness.However, the intrinsic subjectivity of time perceptual thresholds has not been considered yet for the creation of personalised VR environments able to actively measure and manipulate users' perception of time [19].
Capitalising on our previous findings on the existence of the multimodal Kappa effect induced by concurrent visual-tactile stimuli delivered through physical interfaces in the real world, we envisioned that subjective time perception could be distorted through the manipulation of the perceived spatial dimension and the perceived physics of the VR environment [20].Since the effect we found in the real world was mainly driven by vision and was not altered by the integration of concurrent tactile stimulation, in this paper our main focus is on the comparison of the Kappa effect in the unimodal visual and in the concurrent visualtactile stimulation.The need for including also tactile elicitation was motivated by the goal of increasing users' immersiveness and experience in VR, as also highlighted by recent studies [21], [22].If the Kappa effect does not diminish in bimodal stimulation condition also in VR, this can open the path towards the development of multimodal visual-tactile VR interfaces for a more personalised and immersive human-computer interaction.
In this work, we consider, for the first time, the possibility of altering time perception relying on the Kappa effect elicited by concurrent visual-tactile stimuli delivered to the forearm, through a multimodal VR interface (experiment 2), i.e. a wearable haptic device and a virtual visual representation of the users' forearm displayed through a head-mounted display (HMD) (see Fig. 3).The outcomes of experiment 2 are compared with the results of the same experiment performed in the "physical world" (experiment 1), where a multimodal interface was applied to participants' forearm to deliver controlled visual-tactile stimuli.In experiment 1 (i.e. a subset of results of [14]), we measured the Kappa effect in the real-world setup as a reference for V,T,VT modalities.Our main objective is to evaluate if an implementation of the concurrent multimodal visual-tactile stimulation in VR for the elicitation of the Kappa effect produces results that are congruent with what we already found in the physical world.By comparing the Kappa effect in experiment 2 with respect to the Kappa effect found in the real world (experiment 1) we tested three hypotheses: r H1: the Kappa effect can be elicited also in VR r H2: the multimodal integration of spatio-temporal dimen- sions is not altered by VR interfaces wrt the real world r H3: the relation between the ability to discriminate ISIs and the magnitude of the Kappa still holds in VR

II. MATERIAL AND METHODS
The following section shows materials and methods common to both the experiments

A. Hardware
We designed a wearable device, which can deliver tactile and visual stimuli to the forearm.The device (see Fig. 2(a)) has a size of 200 × 22 × 35 mm (L × W × H) and provides five consecutive, evenly spaced stimulation points whose distance was selected with respect to the two-point-discrimination tactile thresholds (i.e., 42 mm [23]).It provides also a visual fixation point placed at the center, at a distance of 24 mm perpendicularly to the stimulation direction.The tactile stimuli were delivered by 5V push-pull solenoids able to tap the skin of the user through a metal tip having a diameter of about 3 mm [24].In experiment 1, the visual stimuli were delivered through a series of W S2812 LEDs (white color) whose positions were in correspondence with those of the solenoids.The onset time and the duration of the stimuli were controlled by an Arduino Mega 2560 microcontroller connected to a laptop PC running Matlab R2021a.The temporal synchronization and stimuli duration were measured by comparing the microcontroller digital output signal of the LEDs with the recordings of an accelerometer (ADXL327Z) with a sampling rate of 800Hz.In experiment 2 (VR), the visual stimuli were delivered through an HMD rendering a white sphere appearing in 5 different locations on a realistic virtual representation of the forearm developed using the Unity framework (see Fig. 3).The HMD was equipped with the Leapmotion to track the position of the hand (freq.120 fps).The temporal synchronization between the vision and somatosensation was calibrated using a light sensor facing the display of the HMD and an accelerometer on repeated measurements.

B. Stimuli and Procedure
Each of the experiments followed a within-subject design in which all participants were tested on all factor combinations of space, complementary ISIs, and modality.The experiments adopted a widely used experimental procedure [25], [26] that provides the observer with three consecutive stimuli (E1, E2, E3), designed to define two ISIs (T 1 , T 2 ) and two contiguous spatial intervals (S 1 , S 2 ), see Fig. 2 a. Independently from the space existing between the stimuli, participants were asked to choose the shortest ISI using a two-alternative forced-choice (2AFC) protocol.
The duration of the stimuli was 47ms [9].The total time T = T 1 + T 2 between the first (E1) and the third (E3) stimulus was always equal to 600ms as well as the total space S = S 1 + S 2 between the first (E1) and the third (E3) equal to 170 mm.[27], [28].The spatial intervals S 1 = S E2−E1 and S 2 = S E3−E2 depended on the factor space, which was labeled based on the spatial distance length of S 1 : short (S), long (L) or equal/control (C).According to the 2AFC protocol, seven combinations of ISI T 1 = T E2−E1 , and T 2 = T E3−E2 with increasing differences were provided to the participants; in one combination the ISI was the same, and equal to 300ms.The minimum difference between ISIs was selected to be 60ms according to the literature [29], and the maximum difference was selected to be 180ms, to ensure sufficient discrimination in both the modalities [30].
Trials were grouped in blocks sharing the factor Modality (i.e., V, T, VT) and fully randomized.In experiment 1 (real-word), all 3 modalities provided 20 repetitions of each factor combination.In experiment 2 (VR), only the V and VT modalities were tested.The motivation for this choice is due to the fact that the tactile stimulation and the related experimental protocol, i.e. the T modality, were the same in real-world and in VR experiments.In other terms, the T condition per se was not influenced by the type of environment (physical, virtual) as the tactile channel was stimulated in the same way.On the contrary, the visual VR modality and the integration of the latter with the tactile stimulation were not tested before.Moreover, in VR each factor combination was repeated 10 times to ensure experimental sessions lasted less than 40 minutes.The stimulation sequence was always elbow-to-wrist as the stimulation direction was proven to not affect the Kappa effect [8].
Experiments took place in a quiet room.Fig. 3 shows a participant wearing the haptic device fastened to the non-dominant forearm (according to the literature [31], the multi-sensory integration is enhanced in the non-dominant hand).Participants were asked to place their arm on the support (tilt 60Â°) keeping their forehead at a distance of about 30 cm perpendicularly to the forearm.
During the experiments, participants wore ear-plugs while a continuous pink noise (approximately 70 dBA) was delivered through earphones to mask any parasitic noise produced by the solenoids.In each trial, participants had to choose the shortest ISI by pressing the left or the right arrows on a keyboard placed nearby the arm support to indicate the first or the second interval, respectively.The next trial started 700−900 ms after a response was recorded.
Before the experiments, participants performed a training phase of 48 trials on all the modalities with complementary ISI of 150 ms and 450 ms to familiarize themselves with the experimental procedure.The difference in the ISIs used in the training phase was greater than the differences tested in the experiment; the experimenter checked that participants were able to distinguish the two intervals and that they clearly understood the task.During the experiments, participants were allowed to rest between blocks at their convenience.

C. Participants
Participants were selected among students and employees of the University of Pisa.They participated on a voluntary basis and were not paid.All of them reported normal or correctedto-normal visual acuity and no sensorimotor impairments.The pool of participants in the two experiments was different.Written Informed consent was obtained prior to participation.The experimental protocol was approved by the Ethical Committee of the University of Pisa (Prot.n. 36590/2021).

D. Data Analysis
For the data analysis, ISI differences were standardized using the Z-scores transform.For each participant, the responses to the 2AFC discrimination task were modeled as a psychometric function for each factor combination of space covered by the first interval (short, equal, long) and modality (V, VT, T) by fitting a General Linear Model (GLM) with a probit link function.We estimated the point of subjective equality (PSE), and the just notable difference (JND).Therefore, for each subject and modality, we computed the magnitude of the Kappa effect as the difference Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.between the PSE of the short (S) and long (L) conditions, which corresponded to the total ISI difference needed to perceive equal ISI as exemplified in Fig. 1.
As individual distributions of PSE and JND were not normally distributed [32], non-parametric group-wise Friedman tests for paired samples [33] were performed.Therefore, whether significant differences emerged, post-hoc pairwise comparisons were performed using non-parametric Wilcoxon test for paired samples with Bonferroni correction [34].

III. EXPERIMENT 1
This experiment tested the Kappa effect in the real-world elicited by V, T and VT stimuli.Fifteen right-handed participants (10 males and 5 females, M = 31.7,SD = 5.9) took part in the experiment.

A. Kappa Effect
The experiment revealed a significant Kappa effect in all modalities showing significant differences across the space factor (p < .0001);post-hoc pairwise comparison showed P SE L < P SE C , P SE C < P SE S and P SE L < P SE S according to the Kappa effect (all p-values < .001).Fig. 4 shows the boxplots of the magnitude of the Kappa effect in all modalities.The Friedman test for paired samples showed significant differences among modalities (p < .005).Post-hoc pairwise comparisons for paired samples revealed significant differences between V and T modalities (p = .01),and between VT and T modalities (p = .02).

B. Duration Discrimination
Fig. 5 reports the boxplot of the JND grouped by modality and space.Separate Friedman tests for paired samples were performed for each factor: in all the modalities the factor space was found not significant.Instead, the factor modality was found significant only concerning the control condition, therefore with equal intervals (p = .002).Post-hoc pairwise comparisons for paired samples revealed significant differences between V and T modalities (p < .0001),and between V and VT modalities (p < .0001).

C. Relation Between JND and Kappa Effect
Fig. 6 reports the scatterplot of the magnitude of the Kappa effect as a function of the mean JND of individual participants calculated separately on each modality.The Spearman's correlations [35] were calculated for each modality finding significant  monotonically increasing coefficients (V R = 0.78, p < .001,T R = 0.80, p < .001,VT R = 0.88, p < .0001).

IV. EXPERIMENT 2
This experiment tested the Kappa effect in VR elicited by V and VT stimuli.Twenty-eight people (20 males, 8 females) aged between 20 and 50 years (M = 25.8,SD = 7.3) took part in the experiment.

A. Kappa Effect
The experiment revealed a significant Kappa effect in both the modalities showing significant differences across the space factor (p < .0001);post-hoc pairwise comparison showed P SE L < P SE C , P SE C < P SE S and P SE L < P SE S according to the Kappa effect (all p-values < .0001).Fig. 7 reports the boxplots of the magnitude of the Kappa effect in V and VT modalities.Differences between V and VT distributions were not significant.

B. Duration Discrimination
Fig. 8 reports the boxplot of the JND grouped by modality and space.Separate Friedman tests for paired samples were performed for each factor: the factor space was significant only in the V modality in which the control equal condition was found significantly higher than the long condition (p = .002).The factor modality was found significant only concerning the   control equal condition in which the JND of the V modality was higher than the VT modality (p < .0001).

C. Relation Between JND and Kappa Effect
Fig. 9 reports the scatterplot of the magnitude of the Kappa effect as a function of the mean JND of individual participants calculated separately on each modality.The Spearman's correlations were calculated for each modality finding significant monotonically increasing coefficients (V R = 0.69, p < .0001,VT R = 0.72, p < .0001).

V. DISCUSSION
In general, the two experiments reported similar outcomes for both the V and VT modalities concerning all the aspects under investigation: the magnitude of the Kappa effect, the duration discrimination thresholds, and their inversely proportional relation.This supports all the working hypotheses H1, H2, and H3.

A. Kappa Effect
A significant Kappa effect was found in all modalities in both experiments.Although smaller than in V and VT modalities, a clear Kappa effect was found in the unimodal tactile condition of experiment 1, confirming the existence of the effect also in contiguous areas of the skin [12], [27].More interestingly, the magnitudes of the Kappa effect found in the real world and in VR were similar in both the V and VT modalities (i.e.median = 1.4,corresponding to a ISI difference of 84 ms), therefore supporting our hypothesis H1.This finding is in agreement with previous literature on duration perception in the real world and in VR [36].Building on the results of the visual-tactile elongated-arm illusion in VR [17] (i.e. the perception of arm size and position can be manipulated through visual-tactile stimulation), we speculate that the range of the magnitude of the Kappa effect may be even further increased in VR by manipulating the perceived size and distance of the forearm.

B. Duration Discrimination
Our second working hypothesis H2 (the multimodal integration of spatio-temporal dimensions is not altered by VR interfaces) is supported by the experimental results.In both experiments, the factor modality was found significant in the control condition (equal space interval) showing an improved time discrimination in the VT modality, in agreement with the literature on the temporal multisensory integration [37].Instead, the same factor was not significant when the Kappa effect was present (S and L conditions).In both experiments, within the VT modality, the factor space was not significant meaning that the position of the stimuli had no effect on the discrimination.

C. Relation Between JND and Kappa Effect
The value of JND and magnitude of the Kappa effect showed a positive correlation in both experiments.Moreover, the linear regression on the data showed an excellent overlap between the V and VT modalities in both experiments confirming our hypothesis H3.In perspective, we speculated that the modeling of the Kappa effect as a function of the JND can be used to estimate the subjective Kappa effect experienced by a user just performing a time discrimination task in a form of a game in VR.

VI. CONCLUSION
To our knowledge, this paper is the first to report on the Kappa effect elicited by visual and concurrent visual-tactile stimuli in VR (experiment 2) in comparison with the same effect found in the real world (experiment 1).Such a comparison revealed similar outcomes in terms of the magnitude of the Kappa effect and duration discrimination in both the V and VT modalities.The Kappa effect and the related alteration of time, which is larger in the visual channel, does not diminish due to Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
bimodal concurrent visual-tactile stimulation both in the real world and in VR.This result opens to the integration of haptic interfaces in VR-based distortion of time perception.Indeed, although not determinant for the elicitation of the illusory effect under investigation, tactile feedback is known to increase users' immersiveness and experience in VR, paving the path towards the development of multimodal visual-tactile VR interfaces for a more personalised and immersive human-computer interaction.In both experiments, the factor modality was significant only in the control condition, where the JND in the V modality is higher than in VT.This is coherent with the Bayesian optimal integration rule described in [38].Finally, the relation between the JND and the magnitude of the Kappa effect found in the real world was confirmed in VR.Larger JNDs seem to be usually associated with a larger Kappa effect.This could open to more personalised manipulation of time perception in VR, where the design of the stimulation paradigm for the elicitation of the Kappa effect will be driven by individuals' time discrimination JND.This could lead to customised stimulation paradigms and richer VR experiences.This point will be further analysed in future works.In the future, we will also investigate the Kappa effect along with body-transfer illusions in VR with the aim of modeling and parameterizing the effect.The results of this paper will pave the path to the effective transferring of the perceived subjective time in VR by exploiting the Kappa effect.

Fig. 1 .
Fig.1.The Kappa effect measured with 3 consecutive stimuli: in the control condition (red) stimuli are equidistant, in the S condition (blue) the space of the first interval is shorter than the second interval, in the L condition (green) the first space interval is longer than the second interval.

Fig. 2 .
Fig. 2. a) schematic of the wearable device reproducing the short (S) and the long (L) space conditions b) temporal synchronization of visual and tactile stimuli measured with a light sensor and an accelerometer.

Fig. 3 .
Fig. 3. Setup of the experiment 2 (VR): a participant wearing the experimental device and the HMD while looking at his forearm.The computer screen shows the participant's view displaying a visual stimulus and the virtual fixation point.
Fig. 2(b) reports the signals of the light sensor and the accelerometer being recorded during a trail simulation; the simulation reproduced the computational load of the experiment while displaying a high contrast scene to properly trigger the light sensor.

Fig. 6 .
Fig. 6.Exp1:scatterplot of the magnitude of the Kappa effect as a function of mean JND of each participant (log scale).

Fig. 7 .
Fig. 7. Exp2:magnitude of the Kappa effect for the V and VT modality.

Fig. 9 .
Fig. 9. Exp2:scatterplot of the magnitude of the Kappa effect as a function of mean JND of each participant (log scale).