Scene Descriptor Expressing Ambiguity in Information Recovery Based on Incomplete Partial Observation

In recent studies, the widespread of deep learning has made many kinds of large-scale image datasets available and it has enabled to improve the performance of image-based 3-D scene reconstruction. Several studies estimate whole 3-D scenes including occluded or unseen parts consistent with the obtained partial observations by integrating prior knowledge from training datasets with them, under no camera parameters nor image landmark correspondence are known. Although they generate “discrete” scene instances, they cannot represent and treat their “ambiguity” at all.This paper proposes a novel deep-learning-based framework that can directly represent and treat the ambiguity of scene reconstructions. We introduce a neural network which encodes a target scene as a descriptor. The network takes partial observations as input and outputs a parametric set of the scene descriptors containing all scenes consistent with given observations. The input observations may be “incomplete” in the sense that they do not have enough pieces of information to uniquely determine the whole scene due to neither geometry in-formation nor landmark correspondences available (ill-defined cases). The network is trained based on the dataset of the complete 3-D scenes and possible partial observations so that it can predict the unseen parts from incomplete observations. The paper introduces the method to induce such a descriptor space into the encoder/decoder architecture by employing novel definitions of loss functions measuring “validity”, “consistency” and “reproducibility”. When the series of partial and incom-plete observations for the same 3-D scene is obtained, the reconstruction ambiguity is explicitly treated by parametrically integrating the descriptor set for each observation into one descriptor set.