Deep 6-DOF Head Motion Prediction for Latency in Lightweight Augmented Reality Glasses

Computationally expensive rendering of virtual 3D objects in mixed reality applications of lightweight AR glasses can be performed by a remotely connected external server. However, nonnegligible 6DOF pose error caused by the remote rendering latency results in 3D visual inconsistency which can be hardly removed by 2D image correction using IMU. In this paper, we propose a novel 6DOF pose prediction algorithm based on learnable combination of consistent motion model and deep prediction. We formulate the combination of both as controlled residual learning and model ensemble. We build a dataset and demonstrate that our algorithm provides accurate prediction under 200ms.