Smart Perception for Situation Awareness in Robotic Manipulation Tasks

Robotic manipulation in semi-structured environments require perception, planning and execution capabilities to be robust to deviations and adaptive to changes, and knowledge representation and reasoning may play a role in this direction in order to make robots aware of the situations, of the planning domains and of their own execution structures. This paper proposes an approach aimed at enhancing the perception capabilities of robotic systems through the integration of various technologies. In particular, the novelties of the proposed smart perception module include the combination of visual sensor data, object detection, and pose estimation techniques, leveraging a fiducial markers and deep learning-based methods, being able to integrate multiple sensors and perception pipelines. In addition, reasoning capabilities are introduced through the utilization of ontologies. The result is a robust and smart perception system capable of handling both simulated and real-world scenarios which in turn provides the required functionalities to allow the robot to understand its surroundings, with a primary focus on robotic manipulation tasks. The discussion on the tools used and the key implementation details are included, as well as the results in some simulated and real scenarios that validate the proposal as a module that provides situation awareness to allow a manipulation framework to adapt the robot actions to uncertain and changing scenarios.