EMS: Energy Minimization Based Video Scene Segmentation

This paper proposes a novel energy minimization based approach to video scene segmentation. In video content analysis, scene is defined as a set of adjacent shots related to a particular setting or a continuous action in one place. This indicates that not only the global distribution of time and content, but also the local temporal continuity should be taken into account for scene segmentation. Motivated from this fact, we formulate the segmentation procedure as a unified energy minimization framework, in which the global and local constraint is represented by content and context energy, respectively. This energy minimization problem is optimized by two steps in an iterative fashion: first find an initial estimation of scene label for content energy by a generative model; and then iterated conditional modes (ICM) is used for context energy to find the global optimization. Furthermore, a boundary voting procedure is devised to decide the optimal scene boundaries. We apply EMS on an extensive set of home videos and feature movies, and report superior performance compared with several existing key approaches to scene segmentation.