Mitigating Cross-Modal Retrieval Violations With Privacy-Preserving Backdoor Learning

Deep cross-modal retrieval, with its effective and efficient search capabilities, has gained widespread adoption in today’s media-sharing practices yet raises concerns regarding potential threats to user data privacy. The cutting-edge data-centric countermeasures usually adopt adversarial learning, i.e., laboriously crafting the proper perturbation for each image, resulting in the noticeable noise in adversarial examples that greatly undermines the aesthetic appeal of image sharing. To address this issue, we propose a novel Model-centric Cross-modal Privacy-preserving framework (MCP), wherein the pre-defined invisible backdoor is seamlessly integrated into the global retrieval model via backdoor learning, thereby effectively preventing shared images containing such triggers from being retrieved. Specifically, we introduce a simple yet effective cross-modal backdoor learning algorithm that alternately optimizes two losses: 1) a privacy-preserving loss for perturbing retrieval with a user-injected trigger and 2) the standard utility loss for maintaining normal retrieval performance. Compared to state-of-the-art methods, MCP excels in providing excellent stealthiness, manifesting in a notable improvement of approximately 100% in SSIM metrics. Furthermore, it achieves an outstanding privacy-preserving (backdoor) success rate, as evidenced by a substantial mAP reduction of 22.3% (for FashionVC), 11.5% (for NUS-WIDE), and 21.8% (for MIRFlickr-25K) in poisoned retrieval, while maintaining similar normal retrieval performance. Additionally, MCP exhibits robust resistance against potential black-box defenses (e.g., trigger filtering) and white-box defenses (e.g., fine-tuning and model pruning). The code and data are available at https://github.com/lqsunshine/MCP.