Space-Warp with Depth Propagation in XR Applications

In this paper, an image guided depth point extraction and depth map reconstruction approach is presented for generating high quality depth maps used for Space-Warp functions on XR devices. The approach can be divided into two parts. The first part is for adaptive depth point extraction to obtain sparse depth points which can efficiently represent the original depth map which runs on device side and the second part is for depth map recovery with the extracted depth points and Space-Warp with the recovered depth map which runs on HMD side. In adaptive depth point extraction, feature points, edges, object contours are extracted as a candidate point set which represents fast depth changes. A criterion is defined by depth map, color image, and spatial information to compute confidence of the candidate points. They are integrated together with the confidence map to build a depth point set. In depth map recovery, an image guided depth map reconstruction from the sparse depth points with depth propagation is presented. During depth map reconstruction, it keeps the sparse points unchanged and propagates the depths in the neighborhood points to the considered points guided by the color image and original depth map information. The reconstructed depth map is used for Space-Warp to create a new frame based on the current frame and a predicted head pose for the new frame. The algorithm is applied to XR systems and applications. The results demonstrate the advantages of the proposed approach. We compare results created by our algorithm and generally used approaches including bilinear interpolation and multilevel B-Spline interpolation. The comparison shows that our algorithm can create high-quality new frames with clear object boundaries and less artifacts.