ODSen: A Lightweight, Real-Time, and Robust Object Detection System via Complementary Camera and mmWave Radar

Deep learning methods have shown promising results in camera-based object detection. However, their effectiveness is significantly hindered in real-world scenes with poor illumination. In contrast, the trend of fusing millimeter-wave (mmWave) radar with camera for object detection is gaining momentum because radar can effectively compensate for the limitations of camera under poor illumination conditions. To this end, we propose ODSen, a lightweight, real-time, and robust fusion detection system. Specifically, ODSen presents several key advantages over existing sensor fusion methods: 1) despite fusing two sensing modalities in a deep learning-based approach, it requires only a small amount of multimodal data for new scenes to achieve real-time and robust object detection; 2) it employs a decoupled architecture that can switch between different image detectors to improve detection accuracy; 3) it performs spatiotemporal fusion of radar and camera features and employs a box refinement model to enhance computational efficiency, thereby ensuring real-time performance without compromising detection robustness. We collect a radar and camera dataset with diverse scenes on a university campus, and conduct extensive experiments, verifying that the proposed ODSen boosts over state-of-the-art methods in terms of average precision (AP), while yielding low computational cost.