DETE: connectionist/symbolic model of visual and verbal association

A description is given of a neurally inspired, computational model of the associative interactions between two types of cognitive functions in humans: the processing of visual and verbal information. DETE, a computer implementation of the model, is designed to learn to associate language descriptions of objects moving in a visual field with those objects, and thus explore the grounding problem, i.e. how language semantics maps to sensory experiences. The model consists of connectionist and microsymbolic modules responsible for processing of the visual and verbal inputs and interacting through an associative module coupled with a mechanism for selective attention. DETE accepts input from two modalities: visual, which consists of a continuous sequence of visual scenes showing the behavior of simple 2-D shaped objects in a square region, and verbal, i.e. occasional streams of English sentences describing the everchanging visual scenes. The overt behavior of the system is language generation and visual imagination.<
>