Photometric image-based rendering for virtual lighting image synthesis

A concept named Photometric Image-Based Rendering (PIBR) is introduced for a seamless augmented reality. The PIBR is defined as image-based rendering which covers appearance changes caused by the lighting condition changes, while Geometric Image-Based Rendering (GIBR) is defined as image-based rendering which covers appearance changes caused by the view point changes. PIBR can be applied to image synthesis to keep photometric consistency between virtual objects and real scenes in arbitrary lighting conditions. We analyze conventional IBR algorithms, and formalize PIBR within the whole IBR framework. A specific algorithm is also presented for realizing PIBR. The photometric linearization makes a controllable framework for PIBR, which consists of four processes: (1) separation of environmental illumination effects, (2) estimation of lighting directions, (3) separation of specular reflections and cast-shadows, and (4) linearization of self-shadows. After the-photometric linearization of input images, we can synthesize any realistic images which include not-only diffuse reflections but also self-shadows, cast-shadows and specular reflections. Experimental results show that realistic images can be successfully synthesized while keeping photometric consistency.