Stereo-based 3D localization for grasping known objects with a robotic arm system

Robust visual servoing is still a difficult problem for a robotic arm system. Within the computer vision community, many object recognition and localization systems have been developed, but in general, they are not tailored to robotic arm applications. In particular, accurate object detection and localization in the camera coordinate system is crucial for grasping. In this paper, we will present a system that makes explicit use of the stereo camera system to locate objects in 3D space and utilizes a commodity graphics card for real-time calculation of depth information that is both low in cost and efficient. In motion planning, the bidirectional RRT (Bi-RRT) algorithm is used to search for a valid path for efficient grasping. Our robot arm can robustly and autonomously grasp single-colored objects and textured objects in 3D space. We realized a grasping scenario with a dexterous arm, ADAM, where an object in front of ADAM could be grasped. This demonstration shows how the proposed components build a dexterous and robust system integrating object detection, stereo vision, and motion planning.