Online Semantic Segmentation and Manipulation of Objects in Task Intelligence for Service Robots

Task Intelligence is the capacity of a robot to learn, reason and execute specific behaviours based on its environment. In this paper, the Task Intelligence problem formulated by the Robot Intelligence and Technology Laboratory at KAIST is researched further: specifically the proposed contribution is a brand new perceptual pipeline in which the recognition, detection, segmentation and grasping of objects is achieved assuming no prior knowledge of the environments arrangement nor the objects appearance. A Convolutional Neural Net (CNN) is used to detect, recognize and semantically label those objects that need to be interacted with. 3D point clouds, corresponding to the objects model, are extracted after several segmentation procedures and registered over time. Dimensional and positional information of the object is acquired. Additional grasping pose data is calculated. All of the collected knowledge is parsed so that the Task Intelligence system is able to deal with previously unknown objects in dynamic environments. This system is formed by an Episodic Memory (Deep-ART), an action sequence generator (FF-planner) and a trajectory warping module for pre-learnt behaviours. The proposed approach has been tested using the Webots simulator.