Parameter-Free Deep Multi-Modal Clustering With Reliable Contrastive Learning

Deep multi-modal clustering (DMC) expects to improve clustering performance by exploiting abundant information available from multiple modalities. However, different modalities usually have heterogeneous distribution with uneven quality. This may lead to limited performance, especially for contrastive multi-modal clustering, which inevitably performs contrastive learning between high-quality and low-quality modalities. To tackle this challenge, we propose a novel framework named parameter-free deep multi-modal clustering with reliable contrastive learning (PDMC-RCL). Specifically, the reliable contrastive learning quantifies the relationship between contrastive modality pairs with weight values that will promote the discriminative features learning from useful modality pairs and slow down or even prevent the learning from unreliable modality pairs. Moreover, the reliable contrastive learning is imposed simultaneously at both the feature-level and cluster-level in this framework so that the feature representation learning can benefit from multi-level contrastive learning. It is worth noting that our PDMC-RCL method is parameter-free, which can achieve promising performance without additional hyperparameter tuning. Experimental results on various datasets show the effectiveness of our method over typical state-of-the-art compared DMCs. The source code is available on https://github.com/ShizheHu