CST: Character State Transformer for Object-Conditioned Human Motion Prediction

Modeling human-object interaction for human motion prediction presents significant challenges due to limited data availability. While pre-training on large-scale datasets offers a promising solution, differences in settings, such as body representations, make it unsuitable for traditional auto-regressive motion predictors. To address this, we introduce the Character State Transformer (CST), a partially pre-trainable model designed for object-conditioned human motion prediction. CST comprises two core components: the task-agnostic Human Joint Transformer (HJT) and the task-specific Spatiotemporal State Transformer (SST). The HJT leverages attention mechanisms to extract spatial features from skeletal structures, while the SST integrates self-and cross-attention to handle multi-modal trajectory and goal-related data. Unlike prior approaches, the HJT can be pre-trained on large motion datasets and fine-tuned for human-object interaction tasks. We also propose a self-supervised masked reconstruction task to enhance HJT pre-training. Experiments on the SAMP dataset demonstrate that CST outperforms existing methods in motion prediction accuracy while requiring fewer parameters. Project page: https://kuan-wei-tseng.github.io/CST.