PhD Forum Abstract: Sensor Fusion for Vehicle-side and Roadside 3D Object Detection and Tracking

We focus on the sensor fusion for vehicle-side and roadside 3D object detection and tracking. Although quite a few sensor fusion algorithms have been proposed, some of which are top-ranked on various leaderboards, a systematic study on how to integrate three crucial sensors (LiDAR, camera and millimeter-wave Radar sensors) to develop effective multi-modal 3D object detection and tracking for vehicle-side perception is still missing. Therefore, we first study the three sensors’ strengths and weaknesses carefully, then compare several different fusion strategies to maximize their utility. Finally, based on the lessons learnt, we propose a simple yet effective multi-modal 3D object detection and tracking framework (namely EZFusion). Without fancy network modules, our proposed EZFusion makes remarkable improvements over the LiDAR-only baseline, and achieves comparable performance. For intelligent transportation, far-range perception with roadside sensors is vital. The main challenge of far-range perception is performing accurate object detection and tracking under far distances (e.g., > 150m) at a low cost. To cope with such challenges, deploying both millimeter wave Radars and high-definition cameras, and fusing their data has become a common practice. Towards this goal, the first question is to conduct the association on the 2D image plane or the BEV plane. We argue that the former is more suitable because the magnitude of location errors in the perspective projection points is smaller at far distances on the 2D plane, leading to more accurate association. Thus, we first project the Radar points to the 2D plane and then associate them with the camera-based 2D object locations. Subsequently, we map the camera-based object locations to the BEV plane through inverse projection mapping (IPM) with the corresponding depth information from the Radar data. Finally, we engage a BEV tracking module to generate target trajectories. Our system is capable of achieving an average location accuracy of 1.3m when we extend the detection range up to 500m.