Multi-Modal Deep Cross-Fusion Network for Autonomous Driving

Due to the complexity of the urban environments, the urban driving task has perennially presented a formidable challenge to autonomous driving. A profound comprehension of the global scenario emerges as pertinent and critical to ensuring road safety. In general, inferring from the global scenario requires manifold types of vehicle-mounted sensor data, each endowing the autonomous system with complementary sensing capabilities. Subsequently, integrating these diverse data sources through data fusion becomes imperative. In previous studies, amalgamating heterogeneous data types, encompassing images and point clouds, has been a customary approach, with the concatenated data being directly fed into the models for subsequent fusion. Nevertheless, it remains an inherent issue that these input features often encompass superfluous and redundant information, potentially leading the autonomous agent to err in its decision-making process, thereby negatively influencing its driving performance. In this work, we proposed the multi-modal cross-fusion imitation learning framework, which is used to learn the spatial representation of the global scenario from the vehicle perspective. We validate the effectiveness of this method in the CARLA urban driving simulator, and our method achieves a driving score of 65.81 on the local test scenario in the Town05 Benchmark.