Estimation of Inter-Pupillary Distance Based on Eye Movements in Virtual Reality Devices

A mismatch between interpupillary distance (IPD) and an inter-optical system distance (IOSD) can lead to various discomforts in playing virtual reality (VR) applications. IOSD must be adjustable to the user’s IPD to solve this issue. This paper investigates IPD estimation methods by tracking eye movements such as conjugate eye movement (CEM) and vergence. We hypothesize that the distance of two pupils is maintained during CEM and it is identical to IPD. The vergence-based method induces eye divergence and determines IPD as the maximum distance between pupils. The experiments with the visual stimuli to induce CEM and divergence are conducted. The average errors of the estimated IPDs for the CEM-based and vergence-based methods are 2.06 mm and 1.30 mm, respectively. If the IOSD is adjusted to the estimated IPD, then VR discomfort issues can be mitigated.


I. INTRODUCTION
Virtual reality (VR) is a computer-simulated environment separated from the real environment, and it could provide a diverse experience that is impossible in the real world [1][2][3]. Most VR environments are mainly implemented with the VR headset such as a head-mounted display (HMD) device. Generally, the HMD devices are composed of a pair of screens to display a different image to each eye. It provides a realistic virtual depth due to the stereoscopic fusion of the images. Fig. 1 shows the major optical factors in the HMD device. Typically, the HMD device contains the optical systems composed of a pair of screens and lenses. An inter-optical system distance (IOSD) is the horizontal interval between two optical systems. The optical systems should be fitted to the individual facial feature, especially, an inter-pupillary distance (IPD) to provide high-quality VR images. IPD is defined as the distance between the centers of the pupils when optical axes of the eyes are parallel to each other [4][5]. If there is a mismatch between IOSD and IPD, the user can suffer from troublesome symptoms during VR experiences such as eye fatigue, disorientation, and visual discomfort [6,7]. Costello [8] found that the misalignment contributes to prismatic distortion, which leads to eye strain and visual discomfort. Lewis and Griffin [9] showed that the discrepancy between IPD and IOSD can make the disruption in accommodation, convergence, and binocular fusion, causing eye strain. Mon-Williams et al. [10] also Inter-optical system distance (IOSD) Optical system Lenses Screens Inter-pupillary distance (IPD) Eyes HMD device showed the discrepancy induces an unstable binocular fusion, and it can be severe to children or adults who already have unstable binocular fusion. Since it results in degrading binocular fusion and visual acuity, the discrepancy may lead to strabismus and diplopia [10]. Furthermore, it could result in visual misperception in the VR environment [11,12]. Utsumi et al. [11] investigated the effect of IPD mismatch on depth perception, and found that it makes the depth of virtual object misperceived significantly. Kim and Interrante studied the influence of the mismatch on perceiving an object's size in the virtual environment [12]. They found that IOSD larger than IPD causes a significant decrease in perceived object size in VR.
According to the previous studies [13][14][15], female with smaller IPD than adult male tends to be more susceptible to those issues. It means that people could have different VR experiences depending on their IPD even if the same VR content is displayed. These uncomfortable feelings can lead to degrading the quality of experience (QoE). Researchers have tried to solve that issue in various aspects [16][17][18][19]. We performed a study focusing on methods for matching IPD and IOSD. Whereas most VR headsets have been designed to have fixed screens and lenses, IPD varies depending on sex, age, race, and even individual [20][21][22][23][24][25][26]. We consider that the non-adjustable IOSD system is a key factor of these problems. Thus, we believe that an adjustable IOSD system would be required in HMD devices. Most conventional HMD devices are composed of the fixed optical systems. Recently, however, a few HMD devices that can "manually" adjust the distance between two lenses are being released such as Oculus Quest [27], HTC VIVE [28], and Sony PlayStation VR [29]. We believe that an automatic adjustment of IOSD by estimating IPD in VR devices will be the next step. Thus, automatic IPD estimation is indispensable.
Several methods have been used in measuring IPD [30][31][32]. The conventional methods, however, require a technician's help for accurate measurement or an extra cost to buy the device such as a pupilometer [30]. Instead, Murray et al. [32] studied measurements of IPD based on the infra-red (IR) camera. Eye locations were derived in 3D space by the camera, and IPD was calculated by the difference between the two eye locations. This method cannot be applied in HMD since it requires a desktop environment and the viewing distance of 60 cm. PlayStation VR provides a software for self-measurement of IPD, but it also requires taking off HMD and additional camera [29]. To overcome these issues, we have studied new methods to estimate IPD accurately by using the VR headset with integrated IR cameras. This paper introduces new IPD measurement methods based on eye movements. Through the experiment, the accuracy and the validity of each method are analyzed. We used IR cameras integrated in the VR headset, called an eye tracker. As various functions using an eye-tracking technique have been studied [18,19,33,34], the eye tracker is becoming one of the major key features needed for future HMDs either to monitor the physical and mental states of the user or to provide more realistic virtual images [35]. We expect that this study will be one of the key functions based on eye-tracking techniques.

A. CONJUGATE EYE MOVEMENT
A conjugate eye movement (CEM) is a movement of both eyes in the same direction to maintain binocular gaze. It is used to either follow a moving object or change the direction of gaze without changing the depth of gaze. From this eye movement, we hypothesized two things: 1. Both eyes would rotate by the same angle in the same direction if the visual stimulus is presented to only one eye. 2. The distance between two eyes would be maintained during CEM as shown in Fig. 2(a) and the distance would be identical to IPD. If so, IPD could be estimated very simply by measuring the distance between two eyes during CEM.

B. EYE VERGENCE
A vergence is in contrast to CEM. It is a simultaneous inward or outward movement of both eyes in opposite directions to maintain single binocular vision. As shown in Fig. 2(b), it includes two types of vergence: convergence, which is the simultaneous inward movement of both eyes toward each other, and divergence, which is the  simultaneous outward movement of both eyes away from each other. This method is based on that the angle between the optical axis and a line connecting eyeball front tops cannot be bigger than 90 degrees during divergence as shown in Fig. 2(b). By tracking the interval between two pupil centers, the maximum value would be identical to IPD.

A. HARDWARE SETUP
A VR headset with integrated IR cameras was used as shown in Fig. 3(a). The cameras recorded the eyes at the speed of 120 frames per second. In order to prevent a screen-door effect and distortion of a visual stimulus by the lens, we removed the lens and display parts in the headset. A pupilometer as shown in Fig. 3(b) was used to measure the participants' true IPDs. A 24-inch LCD monitor whose screen was divided into two parts was used instead of the VR display as shown in Fig. 3

B. VISUAL STIMULI
A white solid square with a size of 1 degree of viewing angle was used as a fixation point. Three types of visual stimuli, as shown in Fig. 4(a)-4(c), were used in the experiment. They are a monocular stimulus (MS), a binocular stimulus (BS), and no stimulus (NS). MS and BS conditions were used to induce the conjugate eye movement and vergence, respectively. Participants can smoothly pursuit the square moving horizontally at a certain speed. During NS condition, no visual object was displayed on the screen. Table 1 describes the conditions of speed for each stimulus type.

C. PARTICIPANTS
Twenty-two observers (female: 6, male: 16) participated in the experiment. Their average age was 25.8 years. We measured every participant's IPD by the pupilometer prior to the experiment. Fig. 5 shows the distribution of the true IPD of the participants. The number of subjects is independent of the order of participation. The red and blue bars denote the results of the females and males, respectively. Detailed statistics are described in Table 2. Large deviation of IPD depending on individual is observed. As with the previous researches, it is observed that the IPD deviation according to individual and sex is larger.

D. PROCEDURE
We let the participant wear the VR headset and measured the distance between the eyes and the lens holes (Deye-lens). Deyelens was used to estimate IPD in the proposed methods. Thereafter, the participants fixed their head on the chinrest and we checked whether the center of the monitor and the VR headset matched. The stimuli were presented in order of MS, BS, and NS. The speed of moving object was sequentially increased for each stimulus. The participants were asked to track the moving square with their eyes while fixing their head. During the trial for NS condition, the participants were allowed to move their gaze freely. Totally seven trials were performed and the experiment was completed within ten minutes for each participant.

E. ESTIMATION OF IPD
The horizontal positions of each eye depending on time when the visual stimulus is presented are shown in Fig. 6(a)-6(c).
In the graph, the y-axis denotes the horizontal position of two pupils in the recorded images and the unit is pixel. The blue and red lines denote the horizontal positions of the right and left eyes, respectively. The black solid line denotes the interval between two pupils. As shown in Fig. 6(a), it is observed that both eyes move in the same direction as expected and the interval is constantly maintained. In the CEM-based method that MS and NS conditions are used, an average value of the interval was estimated as IPD. Fig. 6(b) shows the interval when BS is presented. Since BS induces eye divergence, the interval increases over time as shown in the inner graph in Fig. 6(b). In the vergence-based method, the maximum interval was estimated as IPD.
The estimated IPD was calculated by summing the monocular pupillary distance (mono-PD) of each eye. Mono-PD was estimated through the following procedure. Prior to estimation of mono-PD, a preprocessing to correct perspective distortion was performed as shown in Fig. 7(a). A perspective transform tool for Open CV-Python was used to correct perspective distortion. Since the IR camera aimed slightly upward, the distortion was inevitable. The vertical axes of the image are parallelly aligned after perspective correction, so the horizonal interval would be constant at any y-coordinate. Thereafter, mono-PDs were estimated from the corrected images. The horizontal resolution of the corrected images was 600. The center of the image corresponds to the actual distance of 32 mm away from the center of two IR cameras. As shown in Fig. 7(b) where Px_left and Px_right denote the x-coordinates of the left and right pupil positions, respectively. U is a conversion factor, which is a constant to convert the unit of distance from the pixel in the image to mm, and it depends on Deye-lens.
To find out the relation between U and Deye-lens, we took a picture of a sheet of graph paper and obtained the number of pixels per millimeter under various Deye-lens conditions as shown in the left one of Fig. 7(a). Thereafter, we derived a linear model between U and Deye-lens through regression analysis as the following equation whose coefficient of determination (R 2 ) was 0.99:

IV. RESULTS
To investigate the accuracy of the proposed methods, IPDs estimated by the proposed methods (IPDest) were compared to IPDs measured by the pupilometer (IPDmeter) considered as the true IPD of the participant. An error was calculated by subtracting IPDest from IPDmeter as following equation: error meter est IPD IPD IPD =− (3) We obtained IPDerrors for all conditions and performed a one-way analysis of variance (ANOVA). Fig. 8 shows the obtained data and multiple comparison results. In the chart, the small dots are the data for each participant and the closed curves denote the frequency of them, i.e. density curve. The reddish, bluish, and grayish colored plots denote the results of MS, BS, and NS conditions, respectively. The means and medians are described below each plot. Herein, they were computed in the absolute error. In the MS condition, the errors were, on average, about 2 mm for all speed conditions. The medians for the errors were in the range of 1.89 mm and 2.16 mm. The distribution of IPD errors was from 0 mm to 4 mm, but the result over 4 mm was obtained under the conditions of 8°/s and 12°/s. The accuracy of IPD estimation was the lowest under NS condition. The average and median errors were 2.59 mm and 2.57 mm, respectively. Although the mean IPDerror for NS condition was larger than the IPDerrors for MS condition, no significant difference was observed between them. IPD was estimated with the highest accuracy under BS conditions. The average errors were 1.30 mm, 1.33 mm, and 1.53 mm for 0.2°/s, 0.4°/s, and 0.8°/s conditions, respectively. The errors under BS condition were significantly lower than NS condition. In addition, the error of 0.2°/s was significantly lower than all of MS conditions. Thus, even if it takes more time, BS condition should be applied with a slower speed to get more accurate IPD.
We investigated if the IPD estimation methods can solve the IPD-IOSD mismatch problem in VR environment. We compared the variance of the difference between IPD and IOSD before and after adjusting IOSD. Let's assume that IOSDadj is the IOSD adjusted to the estimated IPD. The IOSD of the VR headset used in this study was fixed to 64 mm (IOSD64mm). Thus, the IPD-IOSD mismatch before adjustment becomes the difference between IPDmeter and 64 mm. After adjustment to IPDest, the IPD-IOSD mismatch corresponds to the difference between IPDmeter and IOSDadj. Fig. 9 shows the distributions of the IPD-IOSD mismatch before and after adjustment. In the graphs, the symbols and the black solid curve denote the IPD-IOSD difference values and their distribution curve, respectively. The mean values before and after adjustment were similar, but the SDs were significantly reduced. For some participants, the IOSD of the VR headset was about 5 mm larger or 13 mm smaller than their own IPDs before adjustment. The SD value was 4.8 mm. After adjustment, however, the IPD-IOSD mismatches were considerably reduced to 0.82-1.02 mm. To analyze the differences of the variance before and after adjustment, Levene test was performed. Table 3 describes the standard deviation and the results of Levene test. Variances for the cases adjusted by the proposed methods with NS, MS, and BS were compared with the variance for the case before adjustment. We found that the proposed methods can significantly reduce the IPD-IOSD mismatch (p<0.05) as shown in Table 3 once we can adjust the IOSD to IPDest obtained by the proposed methods.

V. DISCUSSION
The CEM-based method was used in two stimuli conditions of MS and NS. The results of IPDerror for all participants were over 0 mm, which means that the estimated IPD was smaller than IPD measured by the pupilometer. We speculate that weak eye convergence was maintained during CEM. The weak eye convergence was maintained even in the absence of the visual stimulus. That reveals the weak eye convergence might be natural, resulting in positive IPDerrors for all CEM conditions. We found that the error for MS condition was smaller than that for NS condition, which needs more investigation in our future work. The vergence-based method induced eye divergence until the participant cannot fuse the binocular images. In this reason, we can easily notice the IPD measurement condition in which the optical axes of two eyes are in parallel. Thus, it shows why more accurate IPD could be obtained in the vergence-based method. However, contrary to the CEMbased method, it is observed that a couple of results are negative values as shown in Fig. 9. It shows that, although it is less likely to occur, this method could induce excessive divergence for few users. Kim and Park [7] studied VR sickness depending on the mismatch between IOSD and IPD. They reported that VR sickness critically increases when IOSD is 2 mm larger than the user's IPD, but it does not when IOSD is 2 mm smaller than IPD. Thus, we consider that both proposed methods estimate IPD within the allowable error range in terms of VR sickness. In the case of the CEM-based method, it has the advantage of being easy and fast and does not cause over-divergence, but it has a slightly larger error than the vergence-based method. On the other hand, the vergence-based method could estimate IPD more accurately. Because the maximum eye divergence is induced until they are in parallel, however, it may cause ocular fatigue or difficulty in binocular fusion depending on person.
Although the VR device was used, the experiment was not performed in the VR environment. It is because we tried to minimize the experimental variables such as the uncontrollable screen-door effect and optical distortion. To make the visual stimulus in the VR environment, optical characteristic of the lens must be considered due to the refraction and distortion. Thus, we removed the lens in the VR headset and used the monitor instead of the VR display. We verified that we could obtain IPD very accurately by analyzing eye movements corresponding to visual stimuli with motion. Therefore, we anticipate that our proposed methods will definitely work in VR headset once visual stimuli reflect optical distortion accurately.
The goal of our study is to solve the IPD mismatch problem for improving a VR experience. Most VR headsets including the used one in this study have lenses and displays which are designed for adult males. However, for some people with small IPD such as female, it might be too large to enjoy the VR content without visual discomfort [13][14][15]. According to the previous studies, IOSD larger than IPD can lead to a surprising reduction in QoE for VR [7,36]. To solve that problem, we focused on investigating IPD measurement methods using only VR device as a fundamental research for the auto-IOSD adjustment technique. We verified that the proposed methods can estimate IPD accurately without additional measuring apparatus. We believe that our work will contribute to reducing the negative symptoms caused by IPD mismatch. But there is a limitation that this secondary effect is based on literary inference. To verify it clearly, we will perform our study in the VR environment and evaluate the effectiveness on reducing visual discomfort when using VR devices in future work.

VI. CONCLUSION
This study verified the proposed IPD measurement methods using the IR camera integrated in the VR headset. We investigated the IPD estimation methods based on the eye movements. An experiment with three kinds of visual stimuli to induce two eye movements of CEM and vergence was performed. The average IPDerrors of the CEM-based and vergence-based methods were 2.06 mm and 1.30 mm, respectively. That reveals that both methods can estimate IPD with the allowable accuracy. Furthermore, the analysis results showed that the proposed methods can effectively reduce IPD-IOSD difference and, especially, are helpful for the users with the small IPD.
Our findings suggest that the IR camera integrated in the VR headset could be used for measuring IPD as well as for tracking the gaze. We believe that this study shows a direction for VR headset technology to move forward. Lots of devices have been developed to be personalized, and it will be the same with VR devices. We believe that this study would be a foundation to the technique that adjusts the alignment between IOSD and IPD automatically, and this technique would be a key to solve the problems caused by IPD-IOSD mismatch. Even now, there are people who struggle to play VR content due to small IPD. It is expected that this study will help these people to enjoy VR content without any discomfort. Therefore, we believe that this study