Two-Stream Designed 2D/3D Residual Networks with Lstms for Action Recognition in Videos

Convolutional Neural Networks(CNNs) have achieved great success for object recognition in still images. However, CNNs can't make evident improvement for action recognition in videos, one reason is that many current network architectures are relatively shallow compared with deep models in image domain, and the other reason is that CNNs can't capture effective long-term motion information from videos. Encouraged by the good performance of Residual Network-s(ResNets) for training extremely deep models, and Long-term Recurrent Convolutional Networks(LSTMs) for dealing with tasks involving sequences, we presented an action recognition method based on a two-stream architecture, with 2D ResNets with LSTMs in one stream and designed 3D ResNets with LSTMs in the other stream, which can combine appearance and motion information better. Especially, our proposed method first learns spatiotemporal features of videos through the Residual networks, then models complex temporal dynamics by the Long-term Recurrent Convolutional networks, and with a softmax layer on the top of two streams, the final classification results can be predicted by fusing scores of each stream with weights on score distribution. Furthermore, for better reducing the influence of redundant background information in videos for recognition results, we also applied a center extraction method to generate central regions of videos instead of an entire video into a visual representation. On two video action benchmarks of UCF101 and HMDB51, our method achieved promising performance compared with state-of-the-art.