Spatial–Temporal Attention-Based Human Dynamics Retrospection

Motivated by impressive success of deep recurrent neural networks (RNNs), sequence-to-sequence (seq2seq) architecture has been widely adapted to tackle human motion prediction. However, forecasting in longer time horizons always leads to implausible human poses or converges to mean poses. To address these challenges, we dig into the root causes and lay emphasis on two key principles. First, error can be easily accumulated on seq2seq architecture without modifications and thus RNNs cannot recover from its own mistakes in longer time horizons. Second, all the frames or joints are treated equally, whereas both of them often have different levels of importance in human motion. To mitigate this gap, we propose to retrospect human dynamics with attention. We design a retrospection module built upon seq2seq architecture to recollect previous subsequences and correct mistakes in time which enables a self-correction ability. This assists the original seq2seq architecture to eliminate error accumulation which improves significantly both short-term and long-term performances. Besides, we present two attention techniques to explore correlations among different joints as well as different frames in both spatial and temporal domains, which successfully capture key properties of different actions and enable our model to generate more realistic human poses. Quantitative and qualitative experiments have been both conducted to evaluate the our proposed model. Experimental results clearly demonstrate the superiority of proposed model over other baselines.