A hidden Markov model approach to the structure of documentaries

We have hand-segmented two very long documentaries (100 minutes total) into their component shots. As with other extended videos, shot distribution again appears to be log-normal. Shot lengths are similar to those in dramas, comedies, or action films, but much shorter than those in home videos. The use of fades appears to be an important device to signal transitions between semantic units. We have sought evidence for shot composition rules by means of hidden Markov models (HMMs). We find that camera motion (tilt, pan, zoom) is not significantly governed by rules. However, the bulk of the documentaries take the form of an alternation between commentators and several types of primary supporting material; additionally, the documentaries end with a visual summary. We find that the best approach is one that trains the HMM with labeled subsequences that have approximately equal elapsed time, rather than subsequences with an equal number of shots, or subsequences with shots aligned to some semantic event. This may reflect fundamental temporal limits on human visual attention. We propose that such an underlying structure can suggest more human-sensitive designs for the analysis and graphic display of the contents of extended videos, for summarization, browsing and indexing.