Visual Perception and Modeling for Autonomous Apple Harvesting

Visual perception and modelling are essential tasks in many vision-dependent robotic tasks. This work presents a robotic vision system to perform fruit recognition, modelling, and environment modelling for autonomous apple harvesting. The fruit recognition applies a deep-learning model Dasnet to perform detection and segmentation on fruits, and segmentation on branches. Fruit modelling localises the centre and computes the grasp pose of each fruit based on Hough Transform. Environment modelling adopts Octrees to represent the occupied space within the working environment of the robot. The robot control computes the path and guide manipulator to pick the fruits based on the computed 3D model of the crop. The developed method is tested in both laboratory and orchard environments. Test results show that fruit recognition and modelling algorithm can accurately localise the fruits and compute the grasp pose in various situations. The Dasnet achieves 0.871 on F1 score of the fruit detection. Fruit modelling achieves 0.955 and 0.923 on the accuracy of the fruit centre estimation and grasp orientation, respectively. To illustrate the efficiency of the vision system in autonomous harvesting, a robotic harvesting experiment by using industry robotic arm in a controlled environment is conducted. Experimental results show that the proposed visual perception and modelling can efficiently guide the robotic arm to perform the detachment and success rate of harvesting is improved compared to the method which does not compute the grasp pose of fruits.