Improving Human Action Recognition with Two-Stream 3D Convolutional Neural Network

Human action recognition has became a hot topic in recent years because it opens a wide range of applications such as video surveillance, assisted living, entertainment. Recently, advanced techniques relying on convolutional neural networks produced impressive improvement compared to traditional handcrafted features based techniques. Besides, literature researches also showed that the use of different streams of data will help to increase recognition performance. This paper proposes a method that exploits both RGB and optical flow for human action recognition. Specifically, we deploy a two stream convolutional neural network that takes RGB and optical flow computed from RGB stream as inputs. Each stream has architecture of an existing 3D convolutional neural network (C3D) which has been shown to be compact but efficient for the task of action recognition from video. Each stream works independently then is combined by early fusion or late fusion to output the recognition results. We show that the proposed two-stream 3D convolutional neural network (2stream C3D) outperforms one stream C3D on two benchmark datasets UCF101 (from 82.37% to 88.79%) and HMDB51 (from 48.43 % to 62.54%).