Can Vision Feel Touch? Tactile-aware Visual Grasping for Transparent Objects

Transparent object manipulation has long posed a significant challenge in robotic grasping tasks. Existing methods for transparent object grasping rely heavily on visual sensors, aiming to extract relevant features from raw visual data to facilitate grasp execution. However, transparent objects often possess unreliable visual properties, while tactile contact reliably captures their physical properties. Moreover, these visual-based methods often overlook variations in object standing type (OST) and weight, limiting the precise grasping of transparent objects in different physical states. In contrast, humans naturally form memory associations between visual and tactile information and adjust grip force based on tactile feedback. Building on this foundation, we propose tactile-enhanced visual grasping (TEVG)—a novel method that augments robotic visual capabilities with tactile information to enable precise grasping of transparent objects with unknown OST and weight. The TEVG framework comprises two key components: pre-grasp enhancement (PE) and in-hand enhancement (IE). During the pre-grasp phase, PE embeds tactile features into the visual encoder to predict physical properties in advance, facilitating explicit identification of OST and accurate grasp pose prediction through the tactile-enhanced visual (TEV) encoder. IE enables real-time adaptive adjustment of grasp force during contact manipulation, allowing the system to handle objects with unknown weight effectively. Experimental results on two different robotic platforms demonstrate that TEVG significantly enhances the accuracy and stability of grasping transparent objects. The experiment video and project are publicly available at: https://sites.google.com/view/cvft1.