Overwhelming amounts of surveillance video data are increasingly screwed up the pressure on efficient content-based retrieval and other applications. However, semantic gap exists between the low-level visual signal processing and high-level semantic understanding of the video event. In this paper, we propose an ontology-based content archive and retrieval framework for surveillance videos. Different from the generalized multimedia ontology framework, surveillance domain ontology is first designed as the content description schema, based on which video data is analyzed to form description files in Web Ontology Language (OWL). And then, a web-based semantic retrieval engine, which is compatible with the OWL query API, is developed to provide indexing service. Case study of “walking people” and “car parking” demonstrates that the proposed framework could generate OWL description of a video clip, and reversely locate the information efficiently.