Segmenting image sequences by embedding motion and colour cues within a contextual relaxation scheme

The segmentation of variously moving elements in the images of a film sequence is the basis of a range of digital special effects. Segmentation in film post-production is often carried out laboriously by hand, a method that has a number of drawbacks. Automated segmentation methods are of great benefit to the industry. Such methods must not only determine the boundary between foreground and background with high accuracy but also cope with rapidly moving objects and non-stationary backgrounds. The standard pixel-based maximum a posteriori probability classification technique that distinguishes moving foreground objects from a stationary or motion-compensated background does not contain sufficient contextual information for it to be viable for post-production. An extension to the standard technique is presented that incorporates foreground and background motion residuals with contextual information from spatiotemporal neighbours and also additional evidence provided by the colour distributions of foreground and background. The result shows not only a significant improvement in classification accuracy but also a dramatic reduction in user intervention.