A 2D-3D integrated interface for mobile robot control using omnidirectional images and 3D geometric models

This paper proposes a novel visualization and interaction technique for remote surveillance using both 2D and 3D scene data acquired by a mobile robot equipped with an omnidirectional camera and an omnidirectional laser range sensor. In a normal situation, telepresence with an egocentric-view is provided using high resolution omnidirectional live video on a hemispherical screen. As depth information of the remote environment is acquired, additional 3D information can be overlaid onto the 2D video image such as passable area and roughness of the terrain in a manner of video see-through augmented reality. A few functions to interact with the 3D environment through the 2D live video are provided, such as path-drawing and path-preview. Path-drawing function allows to plan a robot's path by simply specifying 3D points on the path on screen. Path- preview function provides a realistic image sequence seen from the planned path using a texture-mapped 3D geometric model in a manner of virtualized reality. In addition, a miniaturized 3D model is overlaid on the screen providing an exocentric view, which is a common technique in virtual reality. In this way, our technique allows an operator to recognize the remote place and navigate the robot intuitively by seamlessly using a variety of mixed reality techniques on a spectrum of Milgram's real-virtual continuum.