Sequential knowledge-driven scene recognition model

Eye movements are an important aspect of human visual behavior. The temporal and space-variant nature of sampling a visual scene requires frequent attentional gaze shifts, saccades, to fixate onto different parts of an image. Experimental evidence suggests that fixations are often directed towards the most informative regions in the visual scene. We develop a model and its simulation that can select such regions based on prior knowledge of similar scenes. Having representations of scene categories as a probabilistic combination of hypothetical objects, i.e., prototypical regions with certain properties, it is possible to assess the likely contribution of each image region to the successive recognition process. Using conditional probabilities for each region given the scene category, the model can then predict its informative value and initiate a sequential spatial information-gathering algorithm, analogous to an eye movement saccade to a new fixation. This algorithm establishes the most likely scene category for a given image.