Learnable Fusion Semantic Scene Completion: Aggregation of Multi-scale Point Cloud and Image Features

Semantic scene completion (SSC) can express both the semantics and geometry of an entire 3D scene, which contributes a lot to the scene understanding capabilities of the autonomous driving system. Moreover, the generated semantic 3D voxels also depict the specific shapes of objects and enable the system to perceive the obstacles more skillfully. However, due to the sparsity of the LiDAR point clouds and the lack of depth information in the images, it’s challenging to complete the scene from a single modality. The rapid advancements in multi-modality fusion within the field of 3D object detection prompt us to explore its application to the SSC task. In this paper, we present a multi-modality fusion network designed for semantic scene completion named Learnable Fusion Semantic Scene Completion (LFSSC). Specifically, our model integrates the image features and point cloud features on the BEV (bird’s-eye view) with a Learnable Fusion Module (LFM). The combined features are subsequently passed through a Learnable Skip Connection-based Up-sample Network (LSCUN) and a completion head to finish the perception task. Extensive experiments prove that our method achieves competitive performance on the SemanticKITTI dataset.