A Real-time and Robust Visual-based Autonomous Driving Model

In autonomous driving, visual-based urban driving is a challenging task. Considering that the operation of any algorithm takes time, in order to ensure its real-time performance, we propose an end-to-end model that can obtain the control parameters of the current and future four frames. Furthermore, our model includes a new ResNet50-ConvLSTM architecture. To be specific, we first trained a data intensifier through conditional imitation learning using Bird’s Eye View Semantic Segmentation (BEV-seg). The data intensifier could get different control actions (e.g., steering angle and throttle) by changing high-level commands (e.g.,turn left, turn right, follow lane and go straight). Then we use the trained data intensifier to supervise the training of a purely vision-based network. Changing commands while training enables the network to achieve robust driving strategies. Finally, the trained network can get the current and future four frames of control action by inputting the image of the front camera. We evaluated our model in the CARLA simulator. Our result shows that our model achieves better results than the baseline methods on the CARLA original and the NoCrash benchmarks.