The Visual System's Internal Model of the World

The Bayesian paradigm has provided a useful conceptual theory for understanding perceptual computation in the brain. While the detailed neural mechanisms of Bayesian inference are not fully understood, recent computational and neurophysiological works have illuminated the underlying computational principles and representational architecture. The fundamental insights are that the visual system is organized as a modular hierarchy to encode an internal model of the world, and that perception is realized by statistical inference based on such internal model. In this paper, we will discuss and analyze the varieties of representational schemes of these internal models and how they might be used to perform learning and inference. We will argue for a unified theoretical framework for relating the internal models to the observed neural phenomena and mechanisms in the visual cortex.