Unsupervised Video Anomaly Detection by Learning Consistency of Appearance and Motion

Unsupervised video anomaly detection aims to train the model on unlabeled videos containing only normal behavior and identify data instances with abnormal objects or behaviors as anomalies. Abnormal events are typically detected due to their unique appearance and motion features. Existing methods mainly rely on various appearance-motion feature fusion techniques to model the information from both modalities for anomaly detection. However, they overlook the semantic consistency between motion and appearance information, resulting in constrained model performance. To address this limitation, we directly model the relationship of semantic consistency between appearance and motion. Given the varying importance of different pixels, manually designing feature fusion methods to represent their consistency is impractical. So, we designed a two-stream feature fusion module based on cross-attention. This module constructs information fusion between appearance and motion semantics in normal videos, dynamically capturing the inherent appearance-motion semantic consistency. We further believe that, regardless of the perspective, the appearance-motion semantic consistency should be similar. Therefore, this paper introduces a new consistency loss, urging the model to fully model the semantic representation of consistency between appearance and motion in normal data, thereby identifying anomalies with lower consistency. Experimental results on various standard datasets and ablation studies have demonstrated the effectiveness of the proposed method.