Panoramic view reconstruction for stereoscopic teleoperation of a humanoid robot

In recent years, robots have become more commonplace as a method for humans to remotely conduct observations or interactions, with commercial applications ranging from customer service to assembly line work. Virtual and augmented reality have shown much promise as methods to visualize and assist with teleoperation of these robots. However, a number of problems still exist with such bidirectional teleoperator relationships, including the intersensory conflict induced by delay of images received on the teleoperator side of a remote session. This can result in nausea, simulation sickness, and unnatural interaction, especially when stereoscopic cameras are present. As a step towards addressing this problem, we introduce the use of stereo panoramic reconstruction in conjunction with an augmented reality interface to improve both the interaction and the sense of embodiment of a teleoperator working through a robot's perspective. Unlike current methods that send latent streams of the robot's eye-cameras to the user, we view reconstruction that also incorporates direct control of the robot's head. This algorithm uses queued time and angle stamped images to reconstruct the robot's stereoscopic view for the teleoperator, which allows for low-latency viewing and field of view expansion. We then conduct experiments comparing our method against a direct camera feed from the robot's eyes. In addition to measuring latency at every point of the data pipeline, we show that this panoramic representation reduces perceived visual delay and elicits positive user feedback.