An ontology for generating descriptions about natural outdoor scenes

We present an image ontology useful for generating descriptive texts about highly unconstrained natural outdoor images, taken under many different conditions - lighting, varying viewpoints, etc. The ontology pre-defines the visual content we are interested in describing. Unlike other image description techniques, which tend to be purely object-centric, we utilize a holistic scene ontology for description. The primitive units defined by the ontology are extracted from an image via stochastic processes. Similarly, attributes of the units, also specified by the ontology, are evaluated. Binary and tertiary relationships between relevant primitives are also evaluated. The values, attributes and relationships of the primitive units are combined, based on a pre-defined set of production rules, in such a way as to generate rich, descriptive sentences about the image. Evaluation strategies are implemented to quantitatively test the meaningfulness of the generated sentences. Results indicate that the proposed scene ontology aids in generating highly relevant, naturalistic and meaningful sentences describing natural outdoor images.