ECPNet: An Efficient Attention-Based Convolution Network with Pseudo-3D Block for Human Action Recognition

Human action recognition has became an important task in computer vision and has received a significant amount of research interests in recent years. Convolutional Neural Network (CNN) has shown its power in image recognition task. While in the field of video recognition, it is still a challenge problem. In this paper, we introduce a high-efficient attention-based convolutional network named ECPNet for video understanding. ECPNet adopts the framework that is a consecutive connection of 2D CNN and pseudo-3D CNN. The pseudo-3D means we replace the traditional 3 × 3 × 3 kernel with two 3D convolutional filters shaped 1 × 3 × 3 and 3 × 1 × 1. Our ECPNet combines the advantages of both 2D and 3D CNNs: (1) ECPNet is an end-to-end network and can learn information of appearance from images and motion between frames. (2) ECPNet requires less computing resource and lower memory consumption than many state-of-art models. (3) ECPNet is easy to expand for different demands of runtime and classification accuracy. We evaluate the proposed model on three popular video benchmarks in human action recognition task: Kinetics-mini (split of full Kinetics), UCF101 and HMDB51. Our ECPNet achieves the excellent performance on above datasets with less time cost.