Motion Dictionary-Based Approach for Highly Generalizable Text-To-Motion Generation

Despite the continuous updates and iterations of large language models in recent years and their impressive achievements in related tasks, establishing a cross-modal model to solve related tasks has always been a challenge, such as how to build a cross-modal model that uses text to drive human motion. Some attempts have been made to use language models to help complete the text to motion cross-modal task, but the motion data and corresponding language descriptions are relatively limited, and common data augmentation methods struggle to effectively align the augmented semantics or motion frames. Reflecting on this issue, our method finetunes a pre-trained large language model to convert any text into specific instructions in a “motion dictionary”, thereby establishing a relationship between any text and motion instructions. The converted motion instructions are then fed into the subsequent text-motion model to generate corresponding motion results. Extensive experiments demonstrate that our model achieves promising results in any input text-driven motion task.