Learning 6-DoF Fine-Grained Grasp Detection Based on Part Affordance Grounding

Robotic grasping is a fundamental ability for a robot to interact with the environment. Current methods focus on how to obtain a stable and reliable grasping pose in object level, while little work has been studied on part (shape)-wise grasping which is related to fine-grained grasping and robotic affordance. Parts can be seen as atomic elements to compose an object, which contains rich semantic knowledge and a strong correlation with affordance. However, lacking a large part-wise 3D robotic dataset limits the development of part representation learning and downstream applications. In this paper, we propose a new large Language-guided SHape grAsPing datasEt (named LangSHAPE) to promote 3D part-level affordance and grasping ability learning. From the perspective of robotic cognition, we design a two-stage fine-grained robotic grasping framework (named LangPartGPD), including a novel 3D part language grounding model and a part-aware grasp pose detection model, in which explicit language input from human or large language models (LLMs) could guide a robot to generate part-level 6-DoF grasping pose with textual explanation. Our method combines the advantages of human-robot collaboration and LLMs’ planning ability using explicit language as a symbolic intermediate. To evaluate the effectiveness of our proposed method, we perform 3D part grounding and fine-grained grasp detection experiments on both simulation and physical robot settings, following language instructions across different degrees of textual complexity. Results show our method achieves competitive performance in 3D geometry fine-grained grounding, object affordance inference, and 3D part-aware grasping tasks. Our dataset and code are available on our project website https://sites.google.com/view/lang-shape. Note to Practitioners—This paper was motivated by the problem of pragmatic 6 DoF robotic grasping for a robot in a real-world scenario. Different from existing work focusing on task-oriented solutions, our research attempts to consider fine-grained grasping as a general and essential modeling problem (i.e. Point 1: which part to grasp; Point 2: how to grasp the specific part). For that, our method fully considers the scalability and compatibility with subsequently developed functional modules. A two-stage loosely-coupled fine-grained grasping framework (LangPartGPD) is designed, including a 3D part language grounding model and a part-aware 6 DoF grasping pose detection model (Point 1). The former is used to link the symbolic-level grasped part representation to the specific region of an object. The latter is to realize kinematic-level grasping point generation based on the constraint of parts (Point 2). To realize the designed model, we propose a newly large-scale language-pointcloud-grasp dataset (LangSHAPE) to train fine-grained 3D part-level grounding and part-constrained grasping detection models. It is noted that our method is purely concerned with the target object, grasped part, and functionality to realize to model the essential fine-grained graspability, which is an insufficiently explored direction currently and could promote task-level grasping performance. The symbolic-level description of grasping in the form of natural language provides an explainable decision and a user-friendly interface for human-robot collaboration. Experiments in simulations and real-robot settings show our method can effectively enable a robot to make a cognitive decision for fine-grained manipulation. Summarily, our work is a preliminary exploration to link symbolic-level cognitive space to kinematic-level control space directly. In future research, we will further combine more complicated cognitive knowledge from public multimodal knowledge graphs and multimodal LLMs, to build a general robotic cognitive decision system using LangPartGPD as the basic components.