Improved CLIP cross-modal retrieval model for fine-grained interactions

The CLIP based on the direct alignment of the dual-stream structure can effectively increase the speed in the inference stage of cross modal retrieval, and has good retrieval performance. However, the CLIP model only performs cross-modal interaction through global features, ignoring the interaction of fine-grained features, which is not conducive to the alignment of images and texts. We propose a Fine-and-Coarse grained Late Interaction cross-modal Retrieval model FCLIR, which adds fine-grained cross-modal late interaction to CLIP, allowing the model to learn both fine-grained and coarse-grained interactions while maintaining the dual-stream structure. In order to better focus on keywords or patches, a lightweight token screening module is introduced. The model was trained on the MSCOCO and Flick30k datasets and gained good results.