Learning Abstract Representations Through Lossy Compression of Multimodal Signals

A key competence for open-ended learning is the formation of increasingly abstract representations useful for driving complex behavior. Abstract representations ignore specific details and facilitate generalization. Here, we consider the learning of abstract representations in a multimodal setting with two or more input modalities. We treat the problem as a lossy compression problem and show that generic lossy compression of multimodal sensory input naturally extracts abstract representations that tend to strip away modalitiy specific details and preferentially retain information that is shared across the different modalities. Specifically, we propose an architecture that is able to extract information common to different modalities based on the compression abilities of generic autoencoder neural networks. We test the architecture with two tasks that allow: 1) the precise manipulation of the amount of information contained in and shared across different modalities and 2) testing the method on a simulated robot with visual and proprioceptive inputs. Our results show the validity of the proposed approach and demonstrate the applicability to embodied agents.