Semantic-guided Depth Completion from Monocular Images and 4D Radar Data

Depth completion aims to recover dense depth maps from sparse depth measurements. It is a fundamental challenge in computer vision that is faced in numerous applications, such as robotics, UAVs, and autonomous driving. With the recent advancement in hardware, millimeter-wave (mmWave) radar technology is now commonly employed in high-level perception tasks of autonomous driving. However, compared to LiDAR point clouds, mmWave radar point clouds tend to be sparse and include several ghost targets. To address these challenges, we propose a semantic-guided network architecture to perform depth diffusion by utilizing the consistent relationship between semantics and depth. Our method first utilizes semantic features from images to extend the height measurements, and then transforms the global depth completion task into a series of category-specific depth diffusion tasks to learn the semantic-depth priors and accommodate radar sparsity. In addition, a gradient-guided Smooth-Edge loss is developed to explicitly constrain the semantic regional consistency and border discontinuity. Meanwhile, a 4D dataset that features Camera, 4D Radar, and LiDAR measurement in diverse scenes is collected to conduct experiments. Extensive experiments demonstrate the superior performance of our proposed method over existing monocular and fusion-based approaches.