Objectvideo forensics: activity-based video indexing and retrieval for physical security applications

We present an architecture and methodology for high-speed activity-based digital video indexing and retrieval for physical security applications. State of the art computer vision algorithms detect objects in real-time video and determine basic activity information such as object type (human, vehicle, etc.), object trajectory, and interactions with other objects. This information is encoded as a light-weight stream of activity-based meta-data that can be distributed over a network, analysed in real-time to detect security threats, and archived for forensic analyses. To enable flexible forensic analysis, an activity query schema is described that is designed specifically for physical security applications. It is based on a traditional Boolean binary tree approach, but with specific modifications for activity queries. This schema is implemented through an intuitive user interface that enables even unsophisticated users to create complex activity queries. The described mechanism of separating the complex computer vision process from activity queries means that forensic analysis can be achieved at extremely high-speeds - database query time instead of video processing time. Finally a case study is presented concerning the use of this mechanism to optimise the performance of a real-time security system through iterative forensic analysis.