STMotionFormer: Language Guided 3D Human Motion Synthesis with Spatial Temporal Transformer

Inspired by the Transformer's adaptive capability in natural language processing and computer vision domains, and supported by CLIP's strong semantic prior knowledge, we propose a novel generative model named STMotionFormer based on improved Transformer block and CLIP for language-guided 3D skeleton-based human motion synthesis. The skeleton sequence, as a kind of spatial temporal dynamic data, not only contains temporal information but also consists of a natural graph structure by different body parts. We focus on this, propose Temporal Transformer block (T-Former) applies attention mechanism along the temporal dimension to capture the long-term relationship, and propose spatial Transformer block (S-Former) applies attention mechanism in the spatial dimension to aggregate and update joints' spatial features. Therefore, our approach can automatically explore both spatial and temporal information, which can generate skeleton sequence with reasonable graph structure as well as global time relationship. Moreover, powerful semantic prior knowledge of CLIP are injected into our motion-language joint manifold. We evaluate our method on the public KIT motion language dataset that contains manually annotated 3D pose sequences. Experimental results show that our model outperforms the state-of-the-art in terms of APE and AVE respectively. Qualitative visualization results indicate that our model can generate human motions that are more compatible with semantic information even than the GroundTruth.