Point representation augmented to surface reconstruction in image-based VR

In this paper we propose a hybrid representation of environment models by a point representation augmented to geometric representation of surface polygons reconstructed from multiple reference images, through which a real time walkthrough of a complex environment can be achieved. By the method, we start from classification of pixels of the source images into two categories, corresponding respectively to the planar and non-planar surfaces in 3D space. For the pixels corresponding to the planar surfaces, the plane coefficients are reconstructed and all their appearances in the reference images are merged to form uniformly sampled texture images by a comparison of sampling rate and resampling. For the pixels corresponding to the non-planar surfaces, a point representation is applied and the redundant pixels are removed again through sampling rate comparison. The remained pixels are organized by OBB-tree according to their space coordinates. At the same time, the holes that are unable to be captured by all the reference images are pre-filled in the preprocessing phase so that the probability of hole appearance in walkthrough is greatly reduced. Under this hybrid representation, texture mapping and point warping are employed to render the novel views, to take full advantages of the acceleration utility of graphics hardware.