Toward Collision-Aware Robotic Fragile Fruit Grasping: A Sim-to-Real Framework for Perception, Reasoning, and Execution

The collision-aware and damage-less robotic grasping of fragile fruit in stacked scene is challenging but of great importance to postharvest industry. Depth perception and grasp pose estimation are key steps to generate grasping strategy. Improper grasping strategy will lead to collision or damage, and influence the added value and shelf life. In this study, a framework was proposed to solve the above problem, which consists of three stages: Depth perception based on stereo vision, implicit collision-aware grasp pose estimation based on depth information, and grasp execution based on hand eye coordination. The PeachStereo (Sythetic) and PeachGrasp datasets were built in simulation environment to overcome the difficulties in obtaining ground truth in physical world. The stereo matching model 3-D convolution guided and multiscale volume fusion network (CGFNet) and grasp pose estimation model Peach grasping-convolutional neural network (PG-CNN) were developed and trained with sim-to-real method. The performance and generalization ability of our method were validated with experiments in both simulation environment and physical world. In all, our framework provided feasible solution for fragile fruit grasping in stacked scene, which is promising in postharvest industry.