Image Surveillance Assistant

Security watchstanders who monitor multiple videos over long periods of time can be susceptible to information overload and fatigue. To address this, we present a configurable perception pipeline architecture, called the Image Surveillance Assistant (ISA), for assisting watchstanders with video surveillance tasks. We also present ISA1, an initial implementation that can be configured with a set of context specifications which watchstanders can select or provide to indicate what imagery should generate notifications. ISA1's inputs include (1) an image and (2) context specifications, which contain English sentences and a decision boundary defined over object detection vectors. ISA1 assesses the match of the image with the contexts by comparing (1) detected versus specified objects and (2) automatically-generated versus specified captions. Finally, we present a study to assess the utility of using captions in ISA1, and found that they substantially improve the performance of image context detection.