Density-Imbalance-Eased LiDAR Point Cloud Upsampling via Feature Consistency Learning

3D point-cloud upsampling, a crucial perceptual module to help us understand the complex scene and object, aims to generate a high-resolution point cloud given a sparse point set. While considerable attention has been paid to single object point-cloud upsampling, literature on upsampling complex scenes has emerged slowly. Remarkably, few related works target LiDAR-based point clouds, which inherently have a density imbalance problem over distance. This paper proposed LiUpNet, a LiDAR-based point cloud upsampling network. Given a sparse and imbalanced point cloud of a natural and complex scene, LiUpNet extracts robust regional features and generates a uniformly distributed dense point cloud that preserves the fine-grained structural architecture. Specifically, in LiUpNet, an attentive and transformer-based feature extractor is applied to learn the detailed regional representation to model the underlying complex local structure. Also, a novel density-invariant feature consistency loss is introduced to improve the robustness of the learned features against the sparsity changes. Finally, given the regional representations, we append a manifold-based upsampler to super-resolve the regional point clouds up to different scales according to the sparsity and implicitly regresses non-linear 3D scene surfaces. A more uniformly distributed and denser scene point cloud is then achieved with the scheme of dynamic regional upsampling rates. The experimental results on single-object and scene point-cloud upsampling show that LiUpNet outperforms related works qualitatively and quantitatively. Additionally, we demonstrate that the enhanced scene point clouds can efficiently improve the performance of downstream tasks such as point-cloud-based 3D object detection and depth completion.

I. Introduction

Depth sensors such as depth cameras and LiDAR are crucial for autonomous driving tasks. They provide the depth information to form 3D point clouds and serve as a significant perceptual module to help us understand the scene. However, due to the inherent property and the limited number of scan lines, LiDAR point clouds are usually sparse and density imbalanced, which causes a loss in geometric structure. Especially, as shown in Fig. 1(a), the farther the distance is, the sparser the point cloud is. This phenomenon may also result in performance degradation in downstream tasks such as LiDAR-based 3D object detection, LiDAR mapping [1], [2] and LiDAR segmentation [3]. To alleviate the above problems, point cloud upsampling becomes a critical and emergent issue. Fig. 1.

(a) A liDAR-based scene point cloud [4]. (b) A single-object point cloud [5]. (c) An RGB scene image and (d) its sparse LiDAR point cloud. (e) and (f) are the upsampled results of (d) generated by PU-GCN [6] and the proposed LiUpNet.