Semi-automatic video object segmentation in the presence of occlusion

We describe a semi-automatic approach for segmenting a video sequence into spatio-temporal video objects in the presence of occlusion. Motion and shape of each video object is represented by a 2-D mesh. Assuming that the boundary of an object of interest is interactively marked on some keyframes, the proposed method finds the boundary of the object in all other frames automatically by tracking the 2-D mesh representation of the object in both forward and backward directions. A key contribution of the proposed method is automatic detection of covered and uncovered regions at each frame, and assignment of pixels in the uncovered regions to the object or background based on color and motion similarity. Experimental results are presented on two MPEG-4 test sequences and the resulting segmentations are evaluated both visually and quantitatively.