HOPE: A Hierarchical Perspective for Semi-Supervised 2D-3D Cross-Modal Retrieval

With the emergence of AI generated content, cross-modal retrieval of 2D and 3D data has obtained increasing research attention. In practical applications, massive amounts of 2D and 3D data need expensive annotation, which would make labels scarce. Even worse, complicated heterogeneous relationships between 2D and 3D data make the problem more challenging. In this research, we study the problem of semi-supervised 2D and 3D cross-modal retrieval and provide a novel method named Hierarchical Alignment with Ambiguous Pseudo-labeling (HOPE) for this problem. The core of HOPE is to align two modalities in the common space from a hierarchical perspective. Specifically, HOPE not only enforces each sample to approach its respective modality-invariant anchors from an individual view, but also measures both prototypes and distribution for both modalities for discrepancy reduction from a group view. To handle label scarcity with limited error accumulation, HOPE employs two branches of perturbed networks to generate ambiguous candidates, which guides the cross-branch supervision using a margin-based ranking objective. In addition, we retrieve reliable unlabeled samples for each anchor with curriculum learning and class balance, which are added into labeled datasets to clear ambiguity. Extensive experiments on various benchmark datasets validate the superiority of the proposed HOPE.