Integration of deep learning-based object recognition and robot manipulator for grasping objects

Many industrial robots have been applied relatively simple operation that requires repeatable tasks. However, through the rapid development of deep learning with 4^th industrial revolution, it prospects that a role of robots will be extended, and the robots are expected to do task that a human can do. For example, in the service robotics, it is being studied about shelf-stocking and replenishment of many types of products such as food or cleaning rooms at home automatically. In this paper, we integrated object recognition system using deep learning approach with grasping system which has a serial manipulator and gripper for the basis of these future technology. We conduct a bin-picking, the bin-picking is to pick-up some objects and move to bin using an integrated system. In this task, we adopt the Mask R-CNN [3], which is widely used in the field of object segmentation, to determine the kind and shape of an object. After this segmentation on the image, we obtain poses of objects using center of gravity and a simple algorithm for orientation. We assume that the objects exist on not 3D space but a plane, so we only determine the direction of rotation of the object in a plane. Also, in order to position the camera flexibly, a marker is attached to the robot and the transformation between the camera and the robot is registered using this marker. Bin-picking experiments on four household objects took an average of 30 seconds per object. Although this time could be shortened, the speed of the manipulator was limited in consideration of safety. Through these experiments, we were able to verify an effectiveness of this integration which has an object recognition using deep learning and grasping system. On the future work, we aim to complete the 6D pose estimation for sophisticated work.