How visual salience influences natural language descriptions

The model presented is part of a larger project called VITRA (visual translator), where we investigate aspects at the interaction of vision and language. A particular interest lies on the information flow from the analysis of visual data to language generation. We focus on how visual information can be used for grounding descriptions in the environment. In cooperation with the visual perception group of the IIFB at the Fraunhofer Institute at the University of Karlsruhe, we have shown how real world visual data in dynamic environments can be used in natural language descriptions. A model based approach is used for automatically generating 3D representations of the environment. Our current work is related to problems which occur if an agent moves through real or synthetic environments. The agent's task during its movement is to incrementally describe a route from a starting point to a destination by refering to visually obtained objects. In general, the whole complexity of AI research is involved, e.g. control laws for movement, early vision processing, high level vision, naive physics, temporal and spatial reasoning, knowledge representation, planning and language processing. In a first approach, we have implemented a software agent called MOSES who describes a path in a synthetic 3D environment. MOSES can only refer to visually obtained objects (landmarks) in the current situation. Information about the path is extracted from a map by using an incremental path finding procedure.