Soft Contact Simulation and Manipulation Learning of Deformable Objects With Vision-Based Tactile Sensor

Deformable object manipulation is a challenging problem due to its complex deformable properties. With the development of artificial intelligence, learning-based methods have shown outstanding performance in robotic manipulation. Previous works have investigated the manipulation of deformable objects via Reinforcement Learning (RL) in simulation. However, they approximate object deformation with particles, using particle states as observations, which are unavailable in reality. To address these issues, we utilize Vision-Based Tactile Sensors (VBTSs) as the end-effector to manipulate and observe the deformable objects. In this work, we develop a new contact simulation environment for deformable objects, including elastic, plastic, and elastoplastic. We utilize RL strategies and expert demonstrations to train agents in the simulation. Finally, we build a real experimental platform to complete the sim-to-real tasks and robustness testing. Our work introduces an innovative strategy that utilizes high-resolution VBTSs for contact simulation and manipulation of deformable objects. The experimental results show superior performances of deformable object manipulation with the proposed method. Note to Practitioners—The manipulation of deformable objects is a classic and challenging contact-rich robotic task. Compared to rigid objects, deformable objects have higher deformation characteristics, which poses a higher challenge to robot manipulation. As a soft optical sensor, the Vision-based tactile sensor can observe the deformation state of a deformable object while manipulating it, providing a new idea for deformable object manipulation. In this work, we developed a soft contact simulator between the deformable object and the vision-based tactile sensor. The simulator can simulate the contact between the sensors and objects with elastic, plastic, and elastoplastic. Upon this simulator, we proposed a benchmark for deformable object manipulation based on vision-based tactile sensors, including transferable observations, tasks, and expert demonstration systems. Finally, we built a corresponding experimental platform to complete the Sim-to-real and robustness experiments of related tasks. The experimental results show that our method is reliable.