Surface Estimation and Tracking using Sequential MCMC Methods for Video Based Rendering

Video based rendering algorithms attempt to render videos of a scene from an arbitrary viewpoint, given a set of input video sequences taken from several fixed viewpoints. These algorithms require either a dense camera array or some knowledge of scene structure. By applying sequential Markov Chain Monte Carlo (MCMC) methods, we show it is possible to estimate the surfaces visible within a scene, and track them over time, in an efficient manner. Initially, a particle filter is applied across image scale to estimate the surfaces present in a scene at a fixed point in time. Following this, surfaces are tracked over time using a particle filter which takes advantage of both frame-to-frame dependancies, and a hierarchical surface model derived from a multiresolution Gaussian mixture model analysis of the surface data. This time-varying surface model, and the images, are the input for a rendering algorithm which uses a fuzzy z-buffer and projective texturing to generate reconstructions.