Fingers See Things Differently (FIST-D): An Object Aware Visualization and Manipulation Framework Based on Tactile Observations

Planning object manipulation policies based on tactile observations alone is a challenging task due to the multi-factorial variances in the measured point cloud (e.g. sparsity, missing regions, rotation, etc.) and the limited sensory information available through tactile sensing. Nevertheless, the mainstream grasp planners are designed for well-structured point cloud data, and lack the crucial ability to plan grasps in unexplored regions that are common during tactile sampling. Hence, it is crucial to detect the grasp regions from incomplete and unstructured tactile point cloud data. To address this limitation, we propose a novel framework that utilizes a support set of CAD models to augment the tactile observations, and thereby facilitate object recognition, visualization, and developing manipulation policies solely using tactile samples. To cope with the noise and sparsity of tactile observations, we propose uGPIS, a surface reconstruction method that utilizes the occupancy possibility function and the Gaussian Process Regression to recover the underlying surface from tactile point clouds. Then, we complete the partially observed tactile point cloud using the prior knowledge obtained from the support set of full CAD models. This prior information will provide the enriched geometric information that is crucial to determine the grasp regions. Our experimental results on a physical simulation show that our method can successfully combine the prior knowledge from the database to enhance the grasp success rate.