Hard-Label Black-Box Adversarial Attacks for Implicit Scene Interactions

Deep learning models excel in various applications but remain vulnerable to adversarial attacks. Previous adversarial attacks focused on the vulnerability of direct content understanding tasks such as classification and object detection. Implicit interactions play a vital role in intelligent tasks. However, limited attention has been paid to implicit interaction understanding, such as relationships and behaviours. This paper addresses this gap by investigating the vulnerability of implicit interaction understanding in the context of adversarial attacks. Specifically, we introduce a novel adversarial attack task, Interaction Attack (IA), which aims to interfere with scene interaction understanding without affecting direct instance recognition. This task presents unique challenges: (a) interactions are intrinsically tied to scene objects, making independent exploration difficult, and (b) interactions often lack explicit visual cues, complicating direct optimization processes. We propose a novel adversarial attack framework for implicit interactions, named Hard-label Black-box Adversarial Instance Attack (HB-AIA). HB-AIA comprises three key modules, including an Interaction Area Sampling module identifying vulnerable anchors for adversarial instance positioning, an Object Category Search module exploring surrogate categories with higher obfuscation scores for vulnerable anchors, and an Adversarial Instance Generation module crafting adversarial instances with targeted co-occurrence obfuscation to disrupt specific interactions in vulnerable areas. Furthermore, we establish an adversarial attack benchmark based on the Human-Object Interaction task to estimate the vulnerability of implicit interactions. Experiments demonstrate the effectiveness.