AffordStruct: Weakly Supervised Affordance Grounding Based on Spatial Interaction and Knowledge-Aware

Affordance grounding refers to identifying the interactive regions of an object that enable it to perform a specific function, which is essential for effective robot interaction with complex environments. Current methods face challenges of incomplete localization and unclear distinction of affordance regions. To address these challenges, we propose a novel framework called AffordStruct, which draws on human experiential knowledge to explore affordance structures from the perspectives of spatial configuration and semantic relationships. Specifically, we first compute multi-order spatial interactions from the spatial configuration perspective to enhance long-range pixel dependencies, thereby reinforcing the network’s sensitivity to affordance-relevant regions and suppressing irrelevant areas. Besides, we design a pyramid hierarchical chain-of-thought prompting to guide the large language model in reasoning about affordance structural attributes. An attention-based fusion strategy is then employed to perform multimodal fusion with egocentric images that contain only the target objects, enabling the model to better perceive affordance structures at the semantic level. Finally, based on the local knowledge transfer mechanism, the affordance knowledge extracted from the exocentric view containing human-object interactions is transferred to the multimodal egocentric view. Extensive experiments on public datasets and robotic grasping tasks demonstrate that our method outperforms existing state-of-the-art affordance grounding models. Note to Practitioners—This paper was motivated by the problem of incomplete and unclear distinction of affordance areas in affordance grounding. In affordance grounding, affordance knowledge is typically learned by observing human-object interactions, while traditional methods often overlook affordance structural information. This paper proposes a novel method that first explores affordance spatial interactions to preserve affordance structures at the spatial configuration level. Then, a prompting strategy is designed to guide the large language model in reasoning about affordance-related semantic structures. These semantic insights are fused with visual information through multimodal integration to enhance affordance perception from a semantic relationship perspective. Experimental results demonstrate the superiority of our method. Furthermore, we deployed the method on a robot to evaluate its practical applicability. The results show that our approach enables the robotic arm to accurately grasp the functional regions of objects.