Cross-Modal Retriever: Unsupervised Image Retrieval with Text and Reference Images

In our proposed work, we address the challenge of Multi-modal Image Retrieval, focusing on developing a model capable of integrating multi-modal information, such as text and images. The primary objective is enhancing image retrieval accuracy corresponding to a given query, extending the user’s expressive capabilities. We present a transformer-based adaptive aggregation model that utilises a straightforward yet effective fusion mechanism to dynamically integrate information from various modalities. This enables us to learn in an unsupervised way. Our method outperforms existing state-of-the-art (SOTA) models on two public benchmarks, namely FashionIQ and CIRR, and shows promising results in the unsupervised setting. The reference image and user-defined captions help search for the user's required image of interest. The qualitative results also demonstrate the effectiveness of our approach in successfully retrieving the target image. The results obtained show the positive aspects of the proposed system.