Multimodal human centric object recognition framework for personal robots

In this paper we focus on a perception system for cognitive interaction between robots and humans especially for learning to recognize objects in household environments. Therefore we propose a novel three layered framework for object learning to bridge the gap between the robot's recognition capabilities at lower neural level to the higher cognitive level of humans using the weighted fusion of multimodal sources like chromatic, structure and spatial information. In the first layer we propose the grounding of the raw sensory information into semantic concepts for each modality. We obtain a semantic color representation by using SLIC super-pixeling followed by a mapping learned from online images using a PLSA model. This results in a probability distribution over basic color names derived from cognitive linguistic studies. To represent structural information, we propose to cluster the ESF features obtained from Pointcloud data into primitive shape categories. This primitive shape knowledge is learned and expanded from the robot's experience. For spatial information a metric map from the navigation system, demarcated into landmark locations is used. All these semantic representations are compliant with a human's description of his environment and further used in the second layer to generate probabilistic knowledge about the objects using random forest classifiers. In the third layer, we propose a novel weighted fusion of the obtained object probabilities, where the weights are derived from the prior experience of the robot. We evaluate our system in realistic domestic conditions provided at a Robocup@Home setting.