CLIP-Based Multi-level Alignment for Text-based Person Search

Text-based person search (TBPS) refers to the retrieval of target pedestrian from image database based on textual description. How to extract fine-grained discriminative information from image and text and how to conduct cross-modal interaction are two main challenges in this domain. Previous studies typically relied on separately pre-trained unimodal models for feature extraction, and only considered alignment at global level. In this paper, we leverage CLIP's inherent crossmodal semantic learning capabilities, propose a novel CLIP-Based Multi-level Alignment framework (MLA-CLIP) facilitating multi-level alignment. MLA-CLIP simultaneously employs two segmentation networks to extract discriminative feature from both image and text modalities. The effectiveness of our proposed model is validated with the improvements of Rank-1 accuracy ranging from 2% to 8%.