Fusing Onboard Modalities with V2V Information for Autonomous Driving

Status quo autonomous driving mechanisms rely on fusing the multimodal sensing data to integrate the information from onboard units of a vehicle, e.g., lidar, camera, etc., and have yet to consider the information obtained via the inter-vehicle communication, such as the status of neighboring peers. In this paper, we consider to integrate not only the local onboard sensing data, but also the neighboring vehicle information from the vehicle-to-vehicle (V2V) data pipe, which is demonstrated to improve the autonomous driving performance significantly. Specifically, the opportunistic V2V messages are input to a transformer based fusing framework to improve the driving accuracy in both short and long routes in CARLA environment. Unlike previous rule-based mechanisms of dealing with the V2V messages, to the best of our knowledge, the proposed method is the first to integrate the V2V data to the neural network which implicitly induce the waypoints for accurate end-to-end autonomous driving. We conduct extensive experiments, whose results well demonstrate the utility of the V2V information, and can provide useful inspirations for future driving system design.