Recovering Human Mesh from Videos by 2D and 3D Deformable Attentions

Existing methods for 3D human mesh recovery from video rely mainly on Recurrent Neural Networks (RNNs) or Transformers. However, due to the limitations of RNNs in temporal modeling and the dense strategies of traditional attention mechanisms, these methods struggle to efficiently model human motion in videos. To address this issue, we propose a novel method that exploits sparse deformable attention mechanisms, efficiently extracting critical spatio-temporal mesh information from the input videos. Specifically, our method consists of two novel modules: the 3D Deformable Mesh Attention (3D-DMA) module and the 2D Deformable Mesh Attention (2D-DMA) module. The 3D-DMA module adaptively extracts human mesh information by sparsely aggregating the features at varied spatio-temporal locations with attentions, while the 2D-DMA module captures the mesh contexts with the attentions of features at sparsely sampled spatial locations. By fusing the outputs of the two modules, we obtain accurate and smooth human body estimations. Extensive experiments show that our model outperforms previous state-of-the-art methods on three widely used benchmarks.