Scaling perception towards autonomous object manipulation — in knowledge lies the power

Mobile robots operating in a human environment face the challenge of recognizing objects that possess a multitude of different visual characteristics, affordances, and are found in visually challenging scenes. Because of this, perceptual capabilities of such robots need to go beyond detection or categorization of objects, and be able to answer queries not only about where certain objects are located based on their class label, but also about functional properties of these. To achieve an optimal performance, robots need to be aware of their environment, the task that they are to execute, and their perceptual capabilities. Given this knowledge, robotic agents need adequate mechanisms that apply the right method at the right time, in the right situation. In this paper we present a self-adaptive robotic perception system, that acts as a planner for task aware robot manipulation and enables querying on a broad domain. This is done through extending our existing perception framework, ROBOSHERLOCK, with the capability to adapt its perception pipelines based on the query, using knowledge-based reasoning. We will demonstrate the success of the approach, by presenting challenging queries, where the benefits of integrating knowledge processing into perception systems is shown.