Cross-Modal Contrast with Image Jigsaw for Self-Supervised Representation Learning of 3D Point Clouds

Contrastive learning has shown impressive progress for the self-supervision based 3D point clouds feature learning. Based on the distance control in the feature space of positive and negative samples, it can obtain effective point cloud feature representations in a self-supervised manner. However, most existing contrast based point cloud learning methods only consider the feature similarity relationship between samples, (e.g., point cloud, voxel or image), which lack of explicit exploration on point cloud structure. Considering that structure is an important property of point clouds, for better feature learning, we propose a effective cross-modal contrast based method with image jigsaw (CrossCon-Jig) to better learn point cloud representations with both semantic and structural information. Specifically, our method includes intra-modal contrast of point cloud, cross-modal contrast between point cloud and rendered image, and point cloud guided image jigsaw. The intra-modal contrast and the contrast of cross-modal focus on the exploring of invariant and consistent feature representations, and image jigsaw guides the model to explore spatial structure information of point clouds. Extensive experimental tests on 3D object classification and 3D object part segmentation tasks have achieved excellent performance, demonstrating the effectiveness of the proposed method.