Visual servoing for dual arm shaping of soft objects in 3D

We propose a real-time vision-based dual arm controller for manipulating soft objects in three dimensions. Our controller relies on a data-based approach to learn how the object deforms, without any prior knowledge of the shape nor material. We control the two arms with a cooperative task representation, which makes them move together in an absolute frame and/or relatively to one another. This allows object deformation in the relative frame and/or object motion in the absolute frame. Through a short initialization phase for extracting visual features with a Principal Component Analysis on the 3D contour points of the object, we estimate an interaction matrix. This matrix relates the visual features to the robot control inputs for the cooperative task. At each step of the deformation, the process is reiterated with the last data and the interaction matrix is updated along a sliding window. Experiments on a few different objects show the performance of the method in real-time.