Incident Retrieval and Recognition in Video Stream Using Wi-Fi Signal

Retrieving incidents from video stream plays an important role in many computer vision applications. However, most video surveillance system can neither recognize incidents nor support content-based retrieval before the video stream is saved into files. As an emerging type of sensing modality, Wi-Fi signal have the potential to become a signal synchronized with the video stream to perform the incidents detection and recognition. In this work, we simultaneously collect the video stream and the Wi-Fi signal in two surveillance scenarios, and develop a LSTM-based classification model that is able to recognize the incidents in surveillance scenarios. Specifically, we first deploy a video surveillance system in two scenarios to capture the video stream and the synchronized Wi-Fi signal that is very sensitive to environmental changes. Second, an incident detection method based on the entropy change of Wi-Fi signal is proposed to find out the start and end time of the incident in the CSI sequence, thus greatly reducing the computational complexity compared with shot detection in the video stream. Third, the deep network LSTM is adopted to develop an incident recognition model that would be used to classify each size-variable CSI segments into known categories corresponding to the types of the incidents. Fourth, using Wi-Fi signal to locate and recognize incidents in the video stream, we build a quick content-based video retrieval system. Last, the experimental evaluation was performed on a group of real Wi-Fi signal samples. The statistical results shows that the proposed incident detection method is feasible and effective to find out the incidents in video files with an average error of 1.5 s. And the evaluation experiment results demonstrate that the proposed multi-classification model acquires an average value of 0.972, 0.973, 0.985, 0.972 and 0.962 for recall, precision, accuracy, F-1 score and Kappa coefficient, respectively.