Kinect with ROS, interact with Oculus: Towards Dynamic User Interfaces for robotic teleoperation

Teleoperation remains an important aspect for robotic systems especially when deployed in unstructured environments. While a range of research strives for robots that are completely autonomous, many robotic applications still require some level of human-in-the-loop control. In any situation where teleoperation is required an effective User Interface (UI) remains a key component within the systems design. Current advancements in Virtual Reality (VR) software and hardware such as the Oculus Rift, HTC Vive and Google Cardboard combined with greater transparency to robotic systems afforded by middleware such as the Robot Operating System (ROS) provides an opportunity to rapidly improve traditional teleoperation interfaces. This paper uses a System of System (SoS) approach to present the concept of a Virtual Reality Dynamic User Interface (VRDUI) for the teleoperation of heterogeneous robots. Different geometric virtual workspaces are discussed and a cylindrical workspace aligned with interactive displays is presented as a virtual control room. A presentation mode within the proposed VRDUI is also detailed, this shows how point cloud information obtained from the Microsoft Kinect can be incorporated within the proposed virtual workspace. This point cloud data is successfully processed into an OctoMap utilizing the octree data structure to create a voxelized representation of the 3D scanned environment. The resulting OctoMap is then displayed to an operator as a 3D point cloud using the Oculus Rift Head Mounted Display (HMD).