Knowledge Augmentation and Task Planning in Large Language Models for Dexterous Grasping

Dexterous grasping is a critical ability for humanoid robots to interact efficiently with the physical environment. Human beings achieve dexterous grasping through a series of high level cognitive processes including target perception, object recognition, feature estimation, and intuitive reasoning. These processes cooperatively contribute to object understanding and the generation of appropriate grasping strategies. However, the current research focuses on establishing large object datasets to estimate object features and employing learning and planning approaches for task deployment, the exploration of the cognitive aspect of dexterous grasping is limited, especially the role of intuition. This paper addresses this research gap by investigating the cognitive processes in dexterous grasping and presents a cognition based grasping system. The proposed system integrates various cognitive processes to enable dexterous grasping. It gathers object information and estimates missing details using a large language model with common sense. Based on the complemented information, the system learns suitable grasp strategies and intuitively guides their execution. Real-world experiments with a anthropomorphic robot hand demonstrated the performance of the proposed system. By leveraging cognitive processes and utilizing the capabilities of a large language model, The proposed method enhances object understanding, generates effective grasping strategies, and provides guidance for the execution of the grasping strategies.