Discrete Memory Addressing Variational Autoencoder for Visual Concept Learning

A substantial aspect of general intelligence is the ability to summarize basic building blocks from various high-level concepts. Artificial vision systems with such hierarchical property can not only perform accurate reasoning for complex observations, but also learn useful low-level knowledge shared across scenes. To achieve this goal, we propose a discrete memory addressing VAE model (DM-VAE) for explicitly memorizing and reasoning about shared primitives in images. A time-persistence memory module is used to store the learned abstract knowledge and to interact with the generative model. The model decides what to pay attention to at each step, and constructs the primitive library automatically as the learning progresses in a fully unsupervised setting. While performing inference, the model attempts to interpret a new observation as a combination of previously learned elements. We further derive a proper variational lower bound which can be optimized efficiently. We conduct visual comprehension experiments on images and demonstrate that our model is able to search, identify, and memorize semantically meaningful primitive concepts.