VRDriving: A Virtual-to-Real Autonomous Driving Framework Based on Adversarial Learning

Due to the high cost of driving testing and the data collection for autonomous cars in the real world, transferring driving models from the simulator to the real world is significant and urgent. This article proposes a novel virtual-to-real driving (VRDriving) framework based on adversarial learning, which is able to transfer the ability of end-to-end learning models trained by simulation virtual data sets to the real world. The framework consists of three parts: 1) an end-to-end learning-based estimator with the cost-sensitive loss function for driving commands estimation; 2) an encoder for extracting the driving domain features from unlabeled real driving images; and 3) a discriminator for distinguishing whether the virtual driving domain features are real or fake. The above three modules can be trained simultaneously. With such a novel learning framework, the driving domain features encoded from the real images are closer to the driving domain features inside the end-to-end estimator via adversarial learning mechanism. Three experiments show the proposed VRDriving framework performs the state-of-the-art estimation accuracy of the steering command than several baselines and other typical frameworks. Therefore, the VRDriving framework is promising to improve the development efficiency of future autonomous driving.