Keystate-Driven Long-Term Generation of Bimanual Object Manipulation Sequences

Learning to forecast or synthesize bimanual object manipulation sequences has broad applications in assistive robotics and extended reality. Previous methods have several limitations: (1) They can only forecast for short durations as the output deteriorates with longer predictions. (2) They minimize the MSE of fine motions, such as cutting or stirring, which are treated as noise and averaged out, leading to static outputs. (3) They model hand-object contact implicitly, resulting in unrealistic motion where the objects float in the air. We address long-term forecasting degradation by decomposing long sequences of bimanual actions into shorter subsequences defined by keystates, minimizing output quality deterioration. Segmenting sequences into meaningful keystates allows us to treat fine periodic motions as primitives without optimizing their MSE in raw trajectories. We construct a motion dictionary to store representative dynamics for each action category, queried at test time to generate fine motions. Lastly, we improve hand-object contact using a novel neural network that forecasts the pose for objects in motion, while encouraging hand-object contact through generative models for 3D hand grasps. We evaluate our approach on publicly available bimanual manipulation datasets, showing significant improvements over the state of the art.