Predictive Visuo-Tactile Interactive Perception Framework for Object Properties Inference

Interactive exploration of unknown objects' properties, such as stiffness, mass, center of mass, friction coefficient, and shape, is crucial for autonomous robotic systems operating in unstructured environments. Precise identification of these properties is essential for stable and controlled object manipulation and for anticipating the outcomes of (prehensile or nonprehensile) manipulation actions, such as pushing, pulling, and lifting. Our study focuses on autonomously inferring the physical properties of a diverse set of homogeneous, heterogeneous, and articulated objects using a robotic system equipped with vision and tactile sensors. We propose a novel predictive perception framework to identify object properties by leveraging versatile exploratory actions: nonprehensile pushing and prehensile pulling. A key component of our framework is a novel active shape perception mechanism that seamlessly initiates exploration. In addition, our dual differentiable filtering with graph neural networks learns the object–robot interaction and enables consistent inference of indirectly observable, time-invariant object properties. Finally, we develop a N-step information gain approach to select the most informative actions for efficient learning and inference. Extensive real-robot experiments with planar objects show that our predictive perception framework outperforms state-of-the-art baselines and showcases it in three major applications for object tracking, goal-driven task, and environmental change detection.