Segmentation of Aerial Surveillance Video Using a Mixture of Experts

In recent years, aerial video surveillance has proved to be an effective way to collect information for a variety of applications including military operations, law-enforcement activities, disaster management and commercial applications. In order to keep pace with the growing demand for aerial surveillance, a vast amount of aerial video needs to be processed automatically. In particular, objects and regions in aerial videos need to be segmented and labeled to enable other automated video processing, such as event detection, summarization, indexing and high level aerial video understanding. In this paper, we propose a video segmentation algorithm for aerial surveillance video. The algorithm uses a Mixture of Experts (MoE) consisting of a supervised image segmentation algorithm named the Trainable Sequential MAP (TSMAP) segmentation algorithm; the unsupervised mean shift image segmentation algorithm and a moving object detection algorithm. Using domain knowledge from aerial video surveillance, the outputs of the three experts are judiciously combined to generate the final segmentation results. One advantage of our MoE based segmentation algorithm is that it exploits the benefits of each of the experts: the unsupervised segmentation algorithm provides accurate region boundaries; the supervised segmentation algorithm provides semantic labeling for each region using texture information; and the moving object detection algorithm provides a reliable object segmentation using the motion information. We illustrate the efficacy of our approach with aerial surveillance videos containing vehicles, roads, buildings, trees, fields, and shadows.