Semantic Visual Understanding of Indoor Environments: From Structures to Opportunities for Action

We present a two-layer representation of the locally sensed 3D indoor environment. Our representation moves one step forward from capturing the geometric structure of the environment to reason about the navigation opportunities for an agent in the environment. The first layer is the Planar Semantic Model (PSM), a geometric representation in terms of meaningful planes (ground and walls). PSM captures more semantics of the indoor environment than a pure planar model because it represents a richer set of relationships among planar segments. In the second layer, we present the Action Opportunity Star (AOS), which describes the set of qualitatively distinct opportunities for robot action available in the neighborhood of the robot. Our two-layer representation is a concise representation of indoor environments, semantically meaningful to both robot and to human. It is capable of capturing incomplete knowledge of the local environment so that unknown areas can be incrementally learned as observations become available. Experimental results on a variety of indoor environments demonstrate the expressive power of our representation.