Constrained Bipartite Graph Learning for Imbalanced Multi-Modal Retrieval

Real-worldmulti-modal retrieval tasks always encounter modal imbalance scenarios. The scales of instances from different modalities are inconsistent and unaligned with each other. Though several methods alleviate the issue by establishing miscellaneous representations of multi-modal data, they still suffer from difficulties like laborious human-being annotations and complex common-space optimization. In the research, we present Constrained Bipartite Graph Learning (CBGL) for imbalanced correlations, where a size-flexible correlation graph is learned from instances' representations. To guide the graph learning, we take advantage of prior side information, including positive pairs and negative pairs, which readily express intra-modality affinity and inter-modality discrepancy. Accordingly, both positive and negative correlations are propagated over instances, and a similarity graph with satisfactory neighbors is achieved. Benefiting from the probabilistic similarities, a query graph is then naturally constructed that directly achieves multi-modal retrieval. To validate the effect, we build a Music-Video-Image (MVI) dataset in regard to music and images with imbalanced scales. Experimental results reported on MVI dataset and three benchmarks demonstrate our prominent superiority over ten representative competitors in multi-modal retrieval.