Constructing Hierarchical Spatiotemporal Information for Action Recognition

Video action recognition is widely applied in video indexing, intelligent surveillance, multimedia understanding, and other fields. Recently, it was greatly improved by incorporating the convolutional neural network (ConvNet). The features of shadow layers in ConvNet tend to model the apparent and motion of actions, and the features of deep layers tend to represent actions. In this paper, we propose to construct hierarchical information by combining the spatiotemporal features of shadow and deep layers in 3D ConvNet for action recognition. Specifically, we use Res3D to extract spatiotemporal information from different types of layers, and transfer the knowledge learned from RGB to optical flow field. We also propose a Parallel Pair Discriminant Correlation Analysis (PPDCA) to fuse the multiple layers' spatiotemporal information into a compact hierarchal action representation. The experimental results show that there is a good balance between accuracy and dimension in our proposed hierarchical spatiotemporal information, and our method not only outperforms the single layer Res3D methods but also achieves recognition performance comparable to that of state-of-the-art methods.