Towards the Object Semantic Hierarchy

An intelligent agent, embedded in the physical world, will receive a high-dimensional ongoing stream of low-level sensory input. In order to understand and manipulate the world, the agent must be capable of learning high-level concepts. Object is one such concept. We are developing the Object Semantic Hierarchy (OSH), which consists of multiple representations with different ontologies. The OSH factors the problems of object perception so that intermediate states of knowledge about an object have natural representations, with relatively easy transitions from less structured to more structured representations. Each layer in the hierarchy builds an explanation of the sensory input stream, in terms of a stochastic model consisting of a deterministic model and an unexplained “noise” term. Each layer is constructed by identifying new invariants from the previous layer. In the final model, the scene is explained in terms of constant background and object models, and low-dimensional pose trajectories of the observer and the foreground objects. The object representations in the OSH range from 2D views, to 2D planar components with 3D poses, to structured 3D models of objects. This paper describes the framework of the Object Semantic Hierarchy, and presents the current implementation and experimental results.