Multimodal Tactile Perception of Objects in a Real Home

When operating in human environments, such as homes, robots could use tactile sensing to better perceive objects. A challenge that has not been sufficiently addressed is the influence of an object's surroundings on tactile perception. Prior research has focused on perception of objects in laboratory settings. Yet, a number of factors found in homes can affect multimodal tactile sensing. For example, the time-varying thermal characteristics of an object's surroundings, such as sunlight, HVAC, and refrigeration, can affect thermal sensing. Likewise, the placement of an object with respect to other objects and surfaces will affect force sensing and alter the way an object moves when pushed. In order to investigate these and other issues, we had a mobile robot reach out and push 47 different objects found in a real home over a three day period resulting in 1340 pushing episodes. We then characterized the performance of data-driven methods (k-nearest neighbors, support vector machines, hidden Markov models, long short-term memory networks) for a variety of tactile perception problems using the first two seconds of force, thermal, and motion sensing data collected by the robot. We paid particular attention to the ability of these methods to generalize what they have learned to different robot velocities, times of day, and object instances. Our results demonstrate the value of multimodal tactile sensing and data-driven methods for tactile perception from short-duration contact, and also illustrate the great diversity of real-world phenomena relevant to tactile sensing.