Augmented visualization: Observing as desired

Over 20 years have passed since a free-viewpoint video technology has been proposed in which a user's viewpoint can be freely set up reconstructed 3D space of a target scene photographed by multi-view cameras. This technology allows us to capture and reproduce the real world as it recorded. Once we capture the world in a digital form, we can modify the world as augmented reality does (i.e., placing virtual objects in the digitized real world). As oppose to this concept, the augmented world also allows us to see through real objects by synthesizing the backgrounds that cannot be observed in our raw perspective directly. The key idea is to generate the background image using multi-view cameras observing the backgrounds at different positions, and seamlessly overlaying the recovered image in our digitized perspective. In this article, our studies regarding such desired view generation techniques are reviewed, and its future directions and open problems are discussed.