A Robust Multi-Object Tracking Method Based on 4D Millimeter Wave Radar and Monocular Vision Fusion

Multi-object tracking (MOT) is an important component of perception system for autonomous vehicles and mobile robots. The 4D millimeter wave (MMW) radar is a novel and promising sensor for MOT for its capability to provide 3D position data and additional Doppler data (3+1D). However, the MOT methods using 4D MMW radar remain scarce, and existing research commonly disregards the potential simplicity of MOT. Moreover, the application of 4D MMW in MOT is constrained by the noise and sparsity of the radar point clouds. To address the above problems, we propose a robust MOT method with a simple modular structure, which integrates 4D MMW radar and monocular vision. The radar data and vision data are first processed in parallel to obtain object detections separately. The detections of the two sensors are then matched and fused using a proposed association method in the camera domain, which can avoid association ambiguities. Finally, a composite structure, mainly consisting of a detection-to-trajectory association module named Fusion-BYTE and a trajectory management module, is designed to exploit the processed sensor detections and yield tracking results. The proposed method is extensively verified on the public View-of-Delft (VoD) dataset and our unmanned aerial vehicle (UAV) dataset. The experimental results demonstrate that our method achieves superior tracking performance compared to the state-of-the-art single-sensor and multi-sensor fusion MOT algorithms, while maintaining high operational efficiency.