OPEN: Occlusion-invariant Perception Network for Single Image-based 3D Shape Retrieval

Single image-based 3D shape retrieval (IBSR) has attracted appealing academic interests recently, which aims to find the corresponding 3D shape from a shape repository for a given single 2D image. However, state-of-the-art methods neglect the discrepancy in the image domain due to unavoidable occlusion. The occluded image representations acting as noise, may perturb the alignment of the normal 2D representations with the 3D representations, resulting in occlusion-sensitive image-shape retrieval. To tackle this crucial challenge, in this paper, we propose a novel Occlusion-invariant PErception Network (OPEN) to learn occlusion-invariant image representations and image-shape correspondence. Specifically, we propose a hard occlusion example mining strategy to sample a hard image pair. Hereafter, to enforce the consistency between normal and occluded 2D images, we propose an Occlusion-invariant Image Consistency (OIC) based on hard image pairs, which gathers 2D image representations of the same instance while pushing away other 2D image representations. In addition, to prevent the 3D representations from perturbation by the occluded 2D representations, we design an Occlusion-invariant Correspondence Consistency (OCC) based on hard image pairs, which pulls the image-specific 3D shape embedding derived by attention mechanism close to the other 2D image representation of the same instance. The combination of OIC and OCC leads to accurate 2D-3D shape matching in challenging occluded scenarios. Our OPEN outperforms state-of-the-art methods by 6%~11% in terms of Top-1 retrieval accuracy on several representative benchmark datasets.