An overview of recent progress in volumetric semantic 3D reconstruction

This paper gives an overview of a recently proposed method of solving dense 3D reconstruction and semantic segmentation from multiple input images in a joint fashion, i.e. as semantic 3D reconstruction. The formulation is cast as a volumetric fusion of depth maps and pixel-wise semantic classification scores. By posing the two problems as a joint optimization problem, both of the tasks can benefit from the other task's information. This leads to formulations which can reconstruct hidden unobserved surfaces. We give an overview of several papers which describe different ways of modeling the data term from the input data and also works which introduce object shape priors to the formulation. We present the basic convex multi-label formulation on which the method builds and also discuss the relation to other reconstruction algorithms which extract semantically annotated 3D models from images.