Visual monitoring of surface deformations on objects manipulated with a robotic hand

Nowadays dexterous manipulation of rigid objects using a robot hand can be achieved fairly well. However, grasping and manipulating deformable objects is still challenging as the force and tactile sensors which are commonly used in such applications can only provide local information about the deformation at the contact points. In this paper, a vision framework is proposed for 3D visually guided grasping and manipulation of deformable objects. This visual monitoring framework, which uses state-of-the-art computer vision methods, provides a robotic hand system with comprehensive monitoring of the deformable object that it manipulates as it tracks its deformation. Stereoscopic vision is used to detect and track in real time the deformation of non-rigid objects in three dimensions and within a complex environment. The technique is tested successfully in real robotic operation conditions using the Barrett hand. The actual object shape is rendered in the 3D virtual environment of the GraspIt! robotic simulator which also displays the hand configuration.