Fine-Grained Chinese Multimodal Image-Text Retrieval: A Fine Feature Alignment Method Based on CNCLIP

This study proposes the Chinese Fine Alignment in Contrastive Language-Image Pre-training (CFA-CLIP), a fine-grained feature alignment model based on Chinese Contrastive Language-Image Pre-training (CNCLIP), to enhance image-text retrieval performance. The method incorporates three components: (1) a data layer that uses a large language model (LLM) for sentence transformations to increase data diversity while preserving semantics; (2) a feature fusion layer that integrates local features of Masked Autoencoders (MAE) and BERT within CNCLIP's dual-stream framework for improved alignment; and (3) an auxiliary loss layer applying multi-stage contrastive learning and optimizing Image-Text Matching (ITM) loss for hard sample training. Experiments on the MUGE and Flickr30K-CN datasets show that CFA-CLIP achieves significant improvements, particularly in fine-grained retrieval.