The Action Recognition with Multi-Clue Sparse 3-Dimensional Convolutional Networks

The current approaches of deep neural network lack capacity on action recognition, yet ineffective structure for spatiotemporal feature learning, and heavy parameters of architecture are even harsh to optimize. In this paper, we propose multi-clue sparse 3-dimensional convolutional networks(MS3D ConvNet) based on landmark works that aims to make it far known in this field. The MS3D ConvNet enhances channels of input through low, medium, and fast pathways with disparate frames of video, frame sequences considering as temporal clue sampled in terms of variance rate of action. The frames of each pathway are convolved with commonly shared kernels forming cohorts as initial results, and pile up three-way cohorts to the 3D block transmitting to subsequent 3D network. The 3D network is modified by DenseNet architecture with randomly dropping out connections among the layers, and introduces 3D convolutional kernels to deal with temporal information, eventually generating 256-dimension codes of actions. Action recognition employs centers of vector clusters of actions instead of stubborn format, which provide a robust convergence environment. Spontaneously, the MS3D ConvNet fosters capacity of action information and hardly ever sustains heavy amounts of parameters which deserves double profits. This fancy structure also leaves an end-to-end fashion training and a possibility to pre-train parameters. Besides that, we devise tailor strategy based on yolo algorithm to endow disparate factors on pixels in compliance with bounding box frontier, regarded as the means of attention. The MS3D ConvNet has been applied to three public action datasets and a self-defined dataset, which performs advanced steps on accuracy, convergence, and robustness.