Plasti-Former: An Optimal Transport Enhanced Transformer Based Inverse Dynamics Model for Plasticine Shape Manipulation

Plasticine shape manipulation (PSM) is essential in food processing and ceramic production. However, it is challenging since high degrees of freedom of plasticine make it difficult for state representation and planning. Some previous approaches focus on optimizing actions using gradients provided by differentiable particle-based simulators, which still have a noticeable gap compared to real scenarios. In this paper, we use a long roller and present a specific inverse dynamics model (Plasti-Former) to perform PSM. The input to Plasti-Former consists of the provided sub-goals and the current point cloud state of the plasticine; the output is the robot's actions. Plasti-Former comprises a rolling direction prediction module and a rolling position prediction module. In the first module, we integrate optimal transport theory with a PointNet-based model trained with 30k virtual data and 6k actual data to predict the roller's rolling direction θ. In the second module, we predict the start rolling position (x, y) using an attention-based model. Note that the input of this attention-based model is a discretized spatial sequence uniquely designed by us, and the features of each subspace are constructed using the optimal transport vectors. We experimentally evaluate Plasti-Former on six representative sequential plasticine shape manipulation tasks and show that Plasti-Former outperforms state-of-the-art approaches in simu-lation. We also conducted the experiments in the real world, demonstrating a promising sim-to-real effect.