Aerial Video Classification by Integrating Global-Local Semantics in ConvNets

Aerial videos captured by uncrewed aerial vehicles are widely used for remote sensing interpretation due to the dynamic and high-resolution data acquisition at a low cost. Although convolutional neural networks have achieved impressive results for video classification, the large-scale variations of the objects and scenes in aerial videos limit further performance improvement. Compared with generic videos, aerial videos have the characteristics of a large field of view that lead to the great challenges of huge scale variations and complex visual patterns for classification. To address these issues, we investigate multilevel feature representation for aerial video classification, which is cast as a two-stream framework that jointly incorporates global and local semantics with multiscale spatiotemporal ConvNets. First, we propose a multiscale spatio-temporal network consisting of pyramid spatio-temporal (PST) blocks and a multitime-scale decision (MTSD) module. The PST blocks contain multiscale kernels with varying sizes to capture different levels of details, while the MTSD module adaptively aggregates the temporal features across different timescales. Second, we design a two-branch global-local architecture to automatically integrate discriminative features from the holistic contextual information and the highly related key regions, where a spatio-temporal region attention module bridges the two streams to locate the important semantic areas in the video. To verify the effectiveness of our method, we conduct comprehensive experiments on two aerial video classification datasets and achieve state-of-the-art results.