Time-to-Line Crossing Enhanced End-to-End Autonomous Driving Framework

End-to-end autonomous driving approach, which directly maps raw input images to vehicle control commands using a deep neural network, is gaining considerable attention from both academia and industry. Researchers have conducted studies on this subject over the past few years. However, they have focused on designing network architecture, and evaluated performance only with root mean square error (RMSE), which did not account for the temporal dynamics of autonomous driving. In this paper, we propose a time-to-line crossing (TLC) enhanced end-to-end driving framework. The proposed framework is original to the industry in three ways. First, for the fine-labeled training dataset for end-to-end autonomous driving, the TLC based label correction algorithm is applied to reduce the inaccuracies of driver action, which is used as a ground truth. Second, we designed a novel deep neural network model based on bi-directional convolutional long short-term memory (bi-CLSTM) which can sufficiently encode the spatial and temporal features in the input image sequence. Third, in addition to the RMSE evaluation metric, we validated the performance of the end-to-end driving model using TLC, the advanced driving support system (ADAS), and a driver performance indicator. We integrated the proposed framework with existing end-to-end driving models on a full-scale autonomous vehicle in the experimental portion of our study. The results show that the proposed framework is valid and that our network model outperforms the existing models in terms of both RMSE and TLC.