Introspective Perception for Long-Term Aerial Telemanipulation With Virtual Reality

This article presents a novel telemanipulation system to advance aerial manipulation in dynamic and unstructured environments. The proposed system features not only a haptic device, but also a virtual reality (VR) interface that provides real-time 3-D displays of the robot’s workspace, as well as a haptic guidance to its remotely located operator. To realize this, multiple sensors, namely, a LiDAR, cameras, and inertial measurement units (IMUs) are utilized. For processing the acquired sensory data, pose estimation pipelines are devised for industrial objects of known and unknown geometries. We further propose an active learning (AL) pipeline in order to increase the sample efficiency of a pipeline component that relies on deep neural networks (DNNs)-based object detection. All of these algorithms jointly address various challenges encountered during the execution of perception tasks in industrial scenarios. In the experiments, exhaustive ablation studies are provided to validate the proposed pipelines. Methodologically, these results commonly suggest how an awareness of the algorithms’ own failures and uncertainty (“introspection”) can be used to tackle the encountered problems. In addition, outdoor experiments are conducted to evaluate the effectiveness of the overall system in enhancing aerial manipulation capabilities. In particular, with flight campaigns over days and nights, from spring to winter, and with different users and locations, we demonstrate more than 70 robust executions of pick-and-place, force application, and peg-in-hole tasks with the DLR cable-suspended aerial manipulator (SAM). As a result, we show the viability of the proposed system in future industrial applications. The project website can be accessed at https://sites.google.com/view/vr-sam/.