Predictive Display With Perspective Projection of Surroundings in Vehicle Teleoperation to Account Time-Delays

Teleoperation provides the human operator with sophisticated perceptual and cognitive skills in an over-the-network control loop. It gives hope of addressing some challenges related to vehicular autonomy which is based on artificial intelligence by providing a fallback plan. Variable network time-delay in data transmission is the major problem in teleoperating a vehicle. On 4G network, variability of this delay is significant (70-150 ms ping). Due to this, both video streaming and driving commands encounter variable time-delay. This paper presents an approach to provide the human-operator with a forecasted video stream that replicates future perspective of vehicle’s field of view accounting for the delay present in the network. Regarding the image transformation, perspective projection technique is combined with correction given by Smith predictor in the control loop. This image transformation accounts current time-delay and tries to address both issues, time-delays as well as variability. For experiment sake, only frontward field of view is forecasted. Performance is evaluated by performing vehicle teleoperation with real vehicle on street edge-case maneuvers and later comparing the cross-track error with and without perspective projection. Results obtained show improvement in path following tasks.