Spatial Attention with Fine Temporary Synthesis for Action Recognition

This paper proposes an action recognition model named Spatial Attention with Fine Temporal Synthesis (SAFTS) processing in both spatial and temporal dimensions. A non-local attention mechanism in the spatial dimension is designed so that each joint cares about the neighboring nodes and increases attention to far nodes with essential relationships. In the graph partitioning problem, a Mean-field partitioning method is adopted such that the graph nodes are given different weights according to varying distances from the point of interest, which can not only improve the recognition accuracy but also improve the perception range and reduce the number of diffusions, which is equivalent to reducing the number of network layers and the number of parameters. In the temporal dimension, multiple convolutional kernels with different scales are applied to process in parallel, which can subtly identify complex actions. Our proposed SAFTS framework was validated with visualization results in Kinetics-400 and NTU-RGB+D datasets, and compared with the Spatial-Temporal Graph Convolutional Networks (ST-GCN) model, there were improvements in recognition accuracy, number of parameters, and computation, as well as significant reductions in training and testing time.