Vision-Based Proximity and Tactile Sensing for Robot Arms: Design, Perception, and Control

Soft-bodied robots with multimodal sensing capabilities hold promise for versatile and user-friendly robotics. However, seamlessly integrating multiple sensing functionalities into soft artificial skins remains a challenge due to compatibility issues between soft materials and conventional electronics. While vision-based tactile sensing has enabled simple and effective sensor designs for robotic touch, there has been limited exploration of this technique for intrinsic multimodal sensing in large-sized soft robot bodies. To address this gap, this article introduces a novel vision-based soft sensing technique, named ProTac, capable of operating either in tactile or proximity sensing modes. This vision-based sensing technology relies on a soft functional skin that can actively switch its optical properties between opaque and transparent states. Furthermore, this article develops efficient learning pipelines for proximity and tactile perceptions, as well as sensing strategies enabled through the timing activation of the two sensing modes. The effectiveness of the soft sensing technology is demonstrated through a soft ProTac link, which can be integrated into newly constructed or existing commercial robot arms. Results suggest that robots integrated with the ProTac link, along with rigorous control formulation can perform safe and purposeful control actions, which enhances human–robot interaction scenarios and facilitates motion control tasks that are challenging to achieve with conventional rigid links.