LLM-Driven “Coach-Athlete” Pretraining Framework for Complex Text-To-Motion Generation

Advanced text-to-motion models should one by one generate all sub-actions in the complex motion following the given motion description. Existing text-to-motion models have performed well on simple motion descriptions but they usually fail when the motion description involves multiple actions, which indicates the lack of ability to generate sub-actions step-by-step. To solve this problem, we propose a "Coach-Athlete" text-to-motion pretraining framework. In the framework, we employ the large language model (LLM) as the Coach to create incremental text and sub-action sequence pairs from complex text-motion samples. With these data, the Coach guides the Athlete, a text-to-motion model, to predict sub-action sequences in complex motion from short to long. Our approach provides a new way to inject the LLM’s motion decomposition and synthesis capabilities into a small-scale text-to-motion model. Experimental results on the current largest text-to-motion dataset HumanML3D and representative dataset KIT-ML show that our pretraining framework can facilitate performances of several state-of-the-art text-to-motion models, especially for complex motions.