Robotic Material Perception Using Active Multimodal Fusion

Robotic material perception is an extremely important but challenging problem for industrial intelligence. The main difficulties come from the fact that the material properties are difficult to be comprehensibly evaluated by single visual, auditory, or tactile modality. Conventional multimodal fusion methods require collecting all of the multimodal information for a testing sample before the recognition. This is expensive, redundant, and may incur large latency. To tackle this problem, a new active fusion framework for the multimodal material recognition is proposed in this paper. We first adopt the adversarial learning method to obtain the modal-invariant representations to effectively bridge the gap between different modalities and then develop a reinforcement learning method for active modality selection. The developed framework and algorithms are evaluated on a publicly available dataset and show promising material recognition results. The developed framework provides an effective method for industrial material inspection.