Visual Servo Control with YOLOv5 and Kalman Filter for Apple Sorting

Real-time sorting of items on an assembly line is a challenging task to accomplish without causing any stoppages. Visual servoing may enabel to be a highly efficient solution. This paper achieved sorting by utilizing the Dobot Magician manipulator and the ZED2i binocular camera as the hardware framework. The system is capable of performing tasks such as targeted identification, tracking, and grasping of various apples, on the moving conveyor belt. This paper studies a new system for sorting immature apples by binocular camera and manipulator, and applies deep learning and visual servo technology. The YOLOv5 algorithm is used to realizes object detection and positioning in 2D image and multi-thread processing of the image to achieve accurate and rapid positioning of different targets. The kinematic model and hand-eye calibration of the manipulator are used to realize the coordinate transformation between the camera and the manipulator, and the real-time detection and tracking of the target are completed. Finally, kalman filter is used to predict the trajectory of the target and complete the sorting action. The YoloV5 algorithm realizes the precise positioning and maturity identification of apples in 2D images, and the ZED2i binocular vision realizes the 3D spatial positioning of apples. Use visual servo control to guide the manipulator for real-time and precise sorting. This study provides an effective way to improve the level of agricultural automation, reduce labor costs and ensure fruit quality, and has high practical value and broad application prospects.