Fusion of Computer Vision Method and Fin Ray Effect Soft Robotic Gripper for Fruit Harvesting

As the demand for food production increases due to population growth, the agricultural sector has turned to automation to address labor shortages, enhance productivity, and promote sustainability. Autonomous robots are now employed in various tasks, including planting, production forecasting, pest and weed control, and harvesting fruits and vegetables. This article focuses on the use of harvesting robots to automate the collection of fallen fruits, specifically oranges, in orchards. Harvesting these fallen fruits can mitigate pest infestations, reduce the need for pesticides, and provide a source of animal feed. In this study, two new technologies were applied: computer vision and automated robotics, to develop a non-destructive system for fruit harvesting. This system is designed to be easy to implement and economically favorable for use in fruit and vegetable processing lines, guaranteeing fruit detection and picking oranges from the ground. The system was developed using a Raspberry Pi 5 to capture images of the fruit with a PiCamera module 3. An XZ gantry was used to position the gripper within the XZ plane for picking the oranges. One object detection model was developed using the TensorFlow Object Detection API and then converted into TensorFlow Lite to identify oranges using deep learning techniques. The model for detecting oranges achieved a mean Average Precision (mAP) of 98.31%, for Intersection over Union (IoU) [0.5:0.65] and 90.29% for (IoU) [0.65:0.80]. In the end, the results of the field trials were satisfactory: out of 100 trials, 87 resulted in complete captures, in which all three fingers of the gripper grasped the fruit evenly. In 13 cases, the tong grasped the fruit with only two fingers, while the third finger adjusted the fruit.