What's on the Other Side? A Single-View 3D Scene Reconstruction

Robots have limited perception capabilities when observing a new scene. When the objects on the scene are registered from a single perspective, only partial information about the shape of the objects is registered. Incomplete models of objects influence the performance of grasping methods. In this case, the robot should scan the scene from other perspectives to collect information about the objects or use methods that fill in unknown regions of the scene. The CNN-based method for objects reconstruction from a single view utilize 3D structures like point clouds or 3D grids. In this research, we revisit the problem of scene reconstruction and show that scene reconstruction can be formulated in the 2D image space. We propose a new representation of the scene reconstruction problem for a robot equipped with an RGB-D camera. Then, we present a method that generates a depth image of the object from the pose of the camera that is on the other side of the scene. We show how to train a neural network to obtain accurate depth images of the objects and reconstruct a 3D model of the scene observed from a single viewpoint. Moreover, we show that the obtained model can be applied to improve the success rate of the grasping method.