An Online Calibration Method for Microsoft HoloLens

An optical see-through (OST) head-mounted display (HMD) is an appropriate option for an image-guided surgical navigation system (IGS) based on augmented reality (AR). However, the calibration of OST-HMDs has been challenging, since the augmented results can only be observed by the users wearing it. In this paper, we proposed an online calibration method based on the single-point active alignment method (SPAAM) for a HoloLens using a virtual tracking system. The transformation matrices between the world coordinates and the virtual holographic coordinates are calculated using a linear model and decomposed by the singular value decomposition (SVD) algorithm. The surgical workspace is analyzed, and there are nine sampling points for the offline calibration: eight at the vertices and one at the center of a cube of the workspace. 20 groups of data at each sampling point are collected. Five alignments should be implanted in the online calibration. The transformation matrices are compensated for the offset of every use of a HoloLens. To assess the accuracy of the calibration method, in accordance with the RANSAC algorithm, eight groups of additional data at random non-sampling locations are collected. Three different users that are familiar with the calibration procedures performed the calibration. The average distance error of our calibration was below 6 mm, and the rotation error was up to 5°. It is a user-friendly, simple method for the online calibration of OST-HMDs.


I. INTRODUCTION
Augmented reality (AR) is a technology in which a user's view of the real world is enhanced or augmented using additional information generated from a computer model [1].Image-guided Surgical Navigation Systems (IGSs) based on augmented reality (AR) have been applied in laparoscopic surgery, neurosurgery, dentistry, orthopedics and other clinical surgeries in recent years [2]- [9].According to display technologies, augmented reality [9] can be classified into the following forms: projection onto patient [6]- [8], [10], [11], static video display [3]- [5], [13], [15], [16], video seethrough (VST) HMDs [2], [17] and optical see-through (OST) HMDs [12], [14], [17], [18].Although visual models can be projected onto patients [6], the discrepancy between the surgeon's point of view and that of the projector limits the AR accuracy [9].Virtual data can be overlaid on an The associate editor coordinating the review of this manuscript and approving it for publication was Tai-Hoon Kim.
endoscopic video [3].This will divide the surgeon's attention between a separate monitor on one side, and the patient and the endoscope on the other [18].Head-mounted displays (HMDs) are preferred methods for AR techniques.One or two head mounted video cameras on HMDs can provide images of the real world from the view of the users [2].Video is combined with graphic images that are created a scene generator that, blends real with virtual.An optical see-through HMD works by placing optical combiners in front of the user's eyes [19].Users can see the real world directly and virtual images bounced off the combiners from half-silvered monitors.The fact that users have no direct view of the real world using VST-HMDs, unlike OST-HMDs, makes surgeons nervous.In addition, users may suffering from nausea wearing OST-HMDs and would be effectively blinded, if VST-HMDs are powered off.Therefore, OST-HMD is a better option for surgery.
The HoloLens (Microsoft Corporation, Washington), an OST-HMD device that can overlay virtual objects onto the real-world surroundings of the user, has been used in surgical navigation systems [12]- [14].The calibration of the HMD, which obtains the mapping between virtual models and real physical objects, is necessary to display virtual models on the HoloLens during surgery.It is necessary in both VST-HMDs and OST-HMDs.VST-HMDs are relatively easy to calibrate, which can use traditional camera calibration procedures.
It is not practical to capture images as seen by the eye, therefore, the correspondences between points in the real world, points in the HMD display, and locations on the retina cannot be directly measured.Since it is unable to capture the images as seen by eyes, OST-HMDs, are difficult to calibrate [19] In 1994, Oishi and Tachi [20] proposed a shooting-gallery method, in which the head of the user was fixed on a chin rest and the user aimed at targets that were shown at varying distances.Azuma [17] used a bore-sight method in which a crosshair pattern must be aligned with the physical line of sight.The single point active alignment method (SPAAM) that was proposed by Tuceryan et al. [1] and Genc et al. [21] is a common display calibration method for optical see-through HMDs.It asks the users to align a crosshair with a known point in the real world.Movements of users are allowed in the SPAAM.This significantly improves the convenience over the shooting-gallery method and bore-sight method that require the user to remain still during calibration.There are several HMD calibration methods based on SPAAM [18], [22], [23].However, human-computer interaction is needed in SPAAM [1], which will have an impact on the accuracy of the calibration and is a major obstacle for the practical usage of OST-HMDs.Tang et al. [19] evaluated four different variants of the SPAAM display calibration procedures to explore the human factors in the different variants for optical see-through HMDs.Extrinsic body movement should be minimized to keep the head stabilized in the calibration procedures [19].Stylus-mark calibration [24] is recommended in literature [19].However, none of the four procedures can achieve a reliable and accurate result for a naive user.Interaction-free display calibration for optical see-through HMDs is suggested by Itoh and Klinker [25], [26].Since it is a method for a single eye, the procedure should be repeated for both eyes.In addition, an additional eye-tracking camera, which is attached on top of the HMD, would complexify the calibration procedures.To finish the stereo-calibration procedure once, a variant calibration method of the SPAAM was proposed [21].Moreover, Qian et al. [22] suggested that SPAAM may be more suitable for a holographic system like the Microsoft HoloLens, which can offer an immersive 3D experience with a controllable focus cue.He presents two calibration methods of the holographic OST-HMD based on a head-anchored tracking system and a world-anchored tracking system.Since there are cumulative errors among the sensors of HoloLens, there is drift of the holograms over time.A world-anchored tracking system was used in this paper.The magnetic tracker [1] was replaced with a purely vision-based one (FIGURE 1) with one or more visual markers on the surface.In addition, because the precision of an affine model is better than that of an isometric model, and fewer alignments are required with similar precision using an affine transformation than a perspective one, and an affine model would be the best model for the calibration of HoloLens [22].
In surgery, after sterilizing, hands of surgeons can only move in aseptic area.However, the calibration method in reference [22] require users to cover the entire workspace.As the whole calibration process reference is time consuming, a quick calibration method in limited region is necessary for surgery.In this paper, we proposed an online calibration method based on the single point active alignment method (SPAAM) for the HoloLens using a virtual tracking system.The whole calibration process consists of off-line calibration and on-line calibration.The offline calibration, which is implanted at the entire workspace, can be carried out preoperatively at regular intervals.The transformation matrices can be compensated according to the variation of between online use and offline calibration.Although the HoloLens still should be calibrated every time when it is in use, small numbers of alignments at the center of the workspace are required in online calibration.

A. MODEL FOR CALIBRATION
The calibration of HMDs calculates a transformation T (•) that maps 3D points from world coordinates {p} to virtual holographic coordinates {q} [22] The transformation T (•) is assumed to be linear, which can be expressed as follows: where, R represents a 3 × 3 rotation matrix, and P represents a 3 × 1 translational vector.
As shown in FIGURE 2, Micron Tracker is used as a worldanchored tracking device.{m} denotes the Micron Tracker's coordinates system, which is used as the world coordinate system.{e} represents the virtual holographical coordinate system in the view of eye.{c} and {h} are coordinate systems of visual markers on calibration box and holographic display.The calibration of the HoloLens calculates a transformation matrix T e m .We assumed that there is a linear relationship between the virtual space and real space.The transformation matrix T e m can be calculated using the equation T e m = T e h T h m .T e h , which is unknown, should be calculated through Equation (3).
where, T h m and T c m represent the transformation matrices between visual markers on calibration box and holographic display and Micron Tracker, and they are transferred via the UDP.In the calibration procedure, the calibration box is aligned to the dotted frames by the users.T c e , which represents the virtual transformation of the calibration box in eye systems, is given by the designer at certain positons.
If n groups of data are collected, T e h can be calculated using the SVD [27]: To avoid anisotropy in the calibration procedures, the calibration box is designed as a 60 × 60 × 60 mm cube.The pose of the box is easy to recognize.

B. OFF-LINE CALIBRATION OF HOLOLENS
In fact, the relationship between virtual space and real space is un-strict linear, because of the holographic imaging mode of the HoloLens.The projection error will increase when objects are not in the workspace.As shown in FIGURE 3, the visual depth of surgeons and the distance between the navigation objects and the users, ranges from 350 mm to 600 mm, and the minimum vertical and horizontal visual angles, which can contain the working space, are 16 • and 23 • respectively.
Data are collected at the eight vertices and one at the center of the cube (FIGURE 3).T i (i = 1, 2, • • • , 9) represent the calculated transformation matrices at different sampling points p i .The coordinates q 0 in the virtual space of point p 0 in real space can be calculated using the linear interpolation as follows: The average rotation matrix of the box is calculated as follows: There are 9 * n groups of data in the offline calibration procedures, which is time-consuming.

C. ON-LINE CALIBRATION OF HOLOLENS
The transformation matrix T consists of both intrinsic and extrinsic parameters.Although the intrinsic parameters of HoloLens do not change while HoloLens is working.The external parameters of HoloLens is different every time when users wear it, which makes the T variable.Online calibration is to compensate the variation of T between online use and offline calibration.The transformation matrices at the center of the cube T c are intraoperatively calculated.The results of the coordinates q and the rotation matrix are compensated as follows: where T c is the transform matrix at the center of the cube.
If k groups of online data (T c ) i are collected, T c and R c can be calculated as follows:

D. FIRST ORDER BUTTERWORTH FILTER
Noises will cause the tracked position to fluctuate, as shown in FIGURE 4 and TABLE 1.Although the fluctuations in the x-axis and y-axis are small, which are 0.752mm and 0.495mm.The fluctuation in z-axis is large.The large fluctuation will jeopardize the calibration accuracy.A Butterworth filter is used to reduce the noises.The Laplace Transform of it is expressed as follows: The analog filter should be transformed to the digital domain using the bilinear transformation.Differential equation of Equation ( 18) in the digital domain can expressed as follows: where, s denotes a complex variable; u(k), which is the sampling position, is the input value of the filter; y(k) represents the output of the filter; T = 0.05 represents the sampling period, and k is the sampling time.
The positions of the box and the HoloLens that are traced by the Micron Tracker are processed using a Butterworth filter.

A. EXPERIMENTAL SETUP
As shown in FIGURE 2, the user wears the HoloLens and a calibration box is given for the alignment.A Micron Tracker is placed at an appropriate location where the markers on the box and the HoloLens can be tracked simultaneously.The Micron Tracker should be running at the procedures of calibration, and the visual positions of the box and HoloLens are recorded to calculate the Transformation matrices.First, a virtual cube is displayed on the HoloLens and the users try to align vertices and edges of the real box to the virtual cube (FIGURE 5).Once the user is satisfied with the alignment, a pinch gesture is used to confirm the positions of the box and HoloLens (FIGURE 6.b).Next, the virtual cube appears in another location in the HoloLens.Nine groups of data at different sampling locations are collected.
The users should repeat the first and the second procedures 20 times for offline calibration (FIGURE 6.a).The transformation matrices are calculated according to the methods in Section II.A and Section II.A.
Before its practical application, online calibration should be implemented (FIGURE 6.a).Users should align the real box to the virtual cube at the center of workspace 5 times.To evaluate the calibration errors, eight additional groups of data are collected (FIGURE 6.a).In real application, once the HoloLens is calibrated offline at the entire workspace, only the online calibration is required for every usage of HoloLens.

B. DEFINITIONS OF CALIBRATION ERRORS
An OST-HMD calibration cannot be objectively measured, since the superimposed objects can only be observed by users.The evaluation of an OST-HMD calibration has been a challenging topic.Some researchers [21], [28] used a camera to measure the accuracy of the calibration of the OST-HMD.In accordance with RANSAC [29], n groups of additional data at random non-sampled locations are collected to verify the accuracy of the calibration method.If p j (j = 1, 2, • • • , n) represents the tracker coordinates of the random non-sampling points in tests, q j represents their virtual coordinates.Then the distance errors of the transformation E p and the rotation errors of the transformation E R can be calculated as follows [30]: where, rot2axisangle(•) is the angle of the transformation matrices of a rotation axis.

C. DEVELOPMENT OF CALIBRATION SYSTEM
Tracking information is transmitted to the HoloLens via a wireless network.The display software is developed in Unity3D, using the HoloLens AR ToolKit package.

D. RESULTS
The comparison of the OCMH (Online calibration method of HoloLens, OCMH) with different filters was conducted.The comparison between our method and the method that was proby Qian et al. [22] (Calibration with world-anchored tracker, CWAT) was conducted.Three users familiar with the calibration procedures performed the calibration.In our test, eight additional groups of data at random non-sampling  points were collected to verify the accuracy of the calibration method.The alignments times were recorded.The distance errors of the OCMH without a filter range from 1.678 mm to 14.392 mm, with a mean value of 7.305 mm (TABLE 2).The rotation errors of the OCMH without a filter range from 2.284 to 16.194 • , with a mean value of 6.443 • (TABLE 2).The distance errors using the OCMH with a Butterworth filter range from 2.043 mm to 11.682 mm, with a mean value of 6.83 mm (TABLE 2 and FIGURE 7).The rotation errors using OCMH with a Butterworth filter range from 1.959 • to 14.226 • , with a mean value of 5.42 • (TABLE 4 and FIGURE 8).The distance errors using the CWAT with a Butterworth filter range from 2.997 mm to 10.397 mm, with a mean value of 5.84 mm (TABLE 2 and FIGURE 7).Rotation errors using the CWAT with a Butterworth filter range from 1.261 • to 11.149 • , with a mean value of 5.35 • (TABLE 5 and FIGURE 8).The alignments times using the CWAT is approximately 30 min; the alignments times of the offline calibration using the OCMH is approximately 90 min, and the alignments times of the online calibration using the OCMH is approximately 6 min.
The five lines from bottom to top in FIGURE 7 and FIGURE 8 represent the minimum value, the lower quartile value, the median value, the upper quartile value and the maximum value respectively.The '+' in FIGURE 7 and FIGURE 8 represent the average value.

IV. DISCUSSIONS AND FUTURE WORK
The SPAAM [1] is a typical method for OST-HMD calibration.Based on the SPAAM, we proposed an online calibration method for the HoloLens similar to Qian et al. [22], which eliminates the need of tracking a line of sight from the HMD.However, 20 alignments should be implemented to calibrate the holographic OST-HMD, which is a time-consuming task.More importantly, the transformation matrices are assumed to be constant at different locations.We analyzed the valid workspace (the cube space in FIGURE 3) in surgical navigation based on augmented reality using the OST-HMDs.The calibration data are collected at nine sampled positions offline.Linear interpolation is used to calculate the coordinates of virtual space of the non-sampled area.The distance errors and the rotation errors using the OCMH are similar to the CWAT.Although the offline alignments times using the OCMH are much longer than those of the CWAT, the alignments times of the OCMH is much shorter than those of the CWAT, since small alignments at the center of the cube are required for the online calibration.The transformation matrices that are calculated offline are compensated for the offset of every wearing of the HoloLens.Linear interpolation is a simple method, however, it might not be the most appropriate algorithm.In the future, we will study the spatial characteristics of the transformation of the HoloLens to optimize the interpolation model for the workspaces.A non-linear mapping between virtual models and real objects may be established to simulate the projection model of the HoloLens as far as possible.
The evaluation of the accuracy of the calibration is hard to objectively measure, since the procedure depends on the human-computer interaction.Although the measurement method in this paper is less subjective than other methods [19], [21], it is still not objective.The mean distance error and the mean rotation error in this paper are 5.88 mm and 5.42 • respectively.The accuracy of the calibration is not good enough for surgical navigation, but it is good enough for non-surgical applications, the accuracy requirements of which are not high.There are various factors that impact the calibration accuracy of OST-HMDs, such as the errors of the tracking system, the machining errors of the calibration box, the calculation errors of computers, movement in use, and the human factor.A first-order Butterworth filter is applied to reduce the noise of the tracking system.Although the calibration accuracy is improved little, the maximum calibration errors are reduced a lot.However, human factors may still be one of the most important factors in the calibration of OST-HMDs.The principles, and the operative mode of the alignments with respect to their impact on human-computer interaction will be explored in depth.
In navigation applications, the calibrator is fixed to the objects after online calibration.Since the position of calibrator, and the objects in virtual coordinate system can be measured, the position of the virtual model of objects can be calculated.The position of the virtual model can change with the motion of the HoloLens or objects.As shown in FIGURE 9, virtual model of the long bone and the calibrator box are overlaid on the real scene.There are obvious deviation between the long bone and the virtual model, which is about 7∼8mm.Although the calibration online time is reduced, the calibration accuracy in this paper cannot meet the surgical requirement.Moreover, the virtual models appear on the real objects, which will reduce the immersion.In the future, we will improve the calibration method and the display methods to make the fusion of virtual and actual reality more accurate and the user experience more comfortable.

V. CONCLUSION
Existing methods are inappropriate for the online calibration of HoloLens.In this paper, we analyzed the surgical workspace and proposed an online calibration method for the HoloLens using a visual tracking systems.The accuracy of our calibration method was below 6 mm, in terms of the average displacement error.We will improve our method in the future to make the calibration methods more user friendly and more accurate.

APPENDIX CALCULATION OF AEVRAGE TRANSFORMATION MATRIX
If {a i } and {b i } are two sets of 3D points, the transformation matrix between them can be expressed as follows: Here, if {a i } and {b i }(i = 1, 2 • • • , n) are homogeneous coordinates, T is a 4×4 matrix, and a i and b i are 4×1 vectors.If a i and b i are coordinates of 3-D points, T is a 3 × 3 rotation matrix, and a i and b i are 3 × 1 vectors.
We want to find the T n×n that minimizes where E is a 4 × 4 or 3 × 3 unit matrix.Therefore, minimizing 2 is equivalent to maximizing

FIGURE 2 .
FIGURE 2. Transformation for the calibration of the HoloLens.

FIGURE 3 .
FIGURE 3. Linear Interpolation method for the calibration of the HoloLens.

FIGURE 4 .
FIGURE 4. Comparison between the positions of the sample value and filter value.(X1, Y1, Z1) represents the coordinates of point p1, and (X2, Y2, Z2) represents the coordinates of point p2.

FIGURE 6 .
FIGURE 6. Workflow of calibration and evaluation.(a) Work flow experiment.(b) Work flow of alignment.

FIGURE 9 .
FIGURE 9. Navigation results in the cameras of HoloLens.
If a i = T i b i , T i could be measured andT i T t i = E, b i − Tb i ) t (T i b i − Tb i ) = n i=1

TABLE 1 .
Fluctuation of the tracking position.

TABLE 2 .
Errors of the OCMH with different filters.

TABLE 3 .
Distance errors of different methods with a filter.

TABLE 4 .
Rotation errors of different methods with a filter.