Predictive Display for Teleoperation Based on 3D Reconstruction Using Lidar-Camera Fusion

Teleoperation could be used to replace a backup safety driver on autonomous vehicles and could play a valuable role in scaling up deployment of autonomous vehicles. The surrounding environment of the remote vehicle needs to be recreated for the teleoperator using video images received over wireless channels. To handle the significant time delays in receiving remote video data, this paper develops a predictive display system that uses deep learning to estimate the current video display for the teleoperator from old (delayed) camera images. First, old camera and Lidar data are fused to create a 3D reconstruction of the remote vehicle environment using conventional and deep-learning-based algorithms. Then the ego-vehicle’s real-time position and orientation variables are estimated using an extended Kalman filter. Predictive modification of the reconstructed old 3D scene is performed using the ego-vehicle’s estimated new trajectory variables. Deep-learning based image in-painting is used to improve image quality. Furthermore, this paper also introduces a new image comparison metric for evaluating the accuracy of the object detection and localization performance in the predictive display image. Real-world experimental data from the nuScenes and Kitti datasets are used for evaluation of the proposed system. The predictive display images are compared with ground truth images using various image comparison metrics and shown to provide significantly superior performance compared to the actual delayed images received over wireless channels.