Construction of an immersive mixed environment using an omnidirectional stereo image sensor

Recently virtual reality (VR) systems have been incorporating rich information available in the real world into VR environments in order to improve their reality. This stream has created the field of mixed reality which seamlessly integrates real and virtual worlds. This paper describes a novel approach to the construction of a mixed environment. The approach is based on capturing the dynamic real world by using a video-rate omnidirectional stereo image sensor. The mixed environment is constructed of two different types of models: (1) texture-mapped cylindrical 3-D model of dynamic real scenes and (2) 3-D computer graphics (CG) model. The cylindrical 3-D model is generated from full panoramic stereo images obtained by the omnidirectional sensor that uses a pair of hexagonal pyramidal mirrors and twelve CCD cameras. A prototype system has been developed to confirm the feasibility of the proposed method, in which panoramic binocular stereo images of the mixed environment are projected on a cylindrical immersive display depending on user's viewpoint in real time.