Scene Hierarchy: A Missing Piece in Scene Classification

With the ultimate aim of semantic understanding of visual scenes, categorizing them into meaningful groups has been a central task in computer vision for more than two decades. From classical handcrafted features to recent self-learnt features, substantial progress has been made to improve its performance. Failed to be noticed by most solutions proposed so far, a hierarchical structure existing in scene categories, termed as scene hierarchy, is discovered in this paper. We argue that such hierarchy could reveal the latent relations among different visual scenes and further advance the state of the art of scene classification. In particular, we present a data-driven approach to identify the scene hierarchy and demonstrate its effectiveness in assisting scene classification.