ICOP: Image-based Cooperative Perception for End-to-End Autonomous Driving

With cutting-edge sensors and learning algorithms developed for vehicular perception, breakthrough advancements have been made in the perception-based end-to-end autonomous driving in recent years. However, the reliability of autonomous driving systems could be compromised by the vulnerability of perception module to occlusion. To address this issue, the integration of vehicle-to-vehicle communication enabled perception data sharing in the dynamic driving task has been proposed and has yielded notable results, as demonstrated by COOPERNAUT, a cooperative system based on distributed lidar perception. In this paper, we introduce ICOP, an end-to-end driving system based on multi-agent camera cooperative perception, to select sensor sharing nodes and to fuse intermediate image data features for learning a driving policy. In the ICOP system, each agent encodes image information into Bird’s Eye View (BEV) representations individually, and these representations are then transmitted as payloads of V2X (vehicle-to-everything) messages via wireless connection, thus enables capturing global spatial interactions among agents to form comprehensive BEV perception information used for final control decision-making. Supported by our designed mechanism of vehicle-to-vehicle communication and transformer block to achieve acceptable image sensory data size for transmission, the experiments suggest that the proposed cooperative perception driving system achieves better results than lidar-based systems in challenging driving situations compared to prior works.