A Bayesian Treatment of Real-to-Sim for Deformable Object Manipulation

We consider the problem of inferring simulation parameters such that the behavior of an object in simulation and the real world look similar. This
real-to-sim
problem is particularly challenging for deformable objects, where conventional techniques fall short as they often rely on precise state estimation and accurate dynamics models. In this letter, we formulate the real-to-sim problem as probabilistic inference over simulation parameters of deformable objects. Our key idea is in how we define the state space of a deformable object. We view noisy keypoints extracted from an image of the object as samples from the distribution that captures object geometry. We then embed this distribution into a reproducing kernel Hilbert space. A sequence of images of a moving object can then be represented by a trajectory of distribution embeddings in this novel state space for deformables. This allows for a principled way to incorporate noisy state observations into modern Bayesian tools for simulation parameter inference. Despite only using a small set of real-world trajectories, we show how the proposed approach can estimate posterior distributions over simulation parameters, such as elasticity, friction and scale, even for highly deformable objects, such as cloth and ropes.