Multi-Task Learning based Video Anomaly Detection with Attention

Multi-task learning based video anomaly detection methods combine multiple proxy tasks in different branches to detect video anomalies in different situations. Most existing methods suffer from one of these shortcomings: I) Combination of proxy tasks in their methods is not in a complementary and explainable way. II) Class of the object is not effectively considered. III) All motion anomaly cases are not covered. IV) Context information is not engaged in anomaly detection. To address these shortcomings, we propose a novel multi-task learning based method that combines complementary proxy tasks to better consider the motion and appearance features. In one branch, motivated by the abilities of the semantic segmentation and future frame prediction tasks, we combine them into a novel task of future semantic segmentation prediction to learn normal object classes and consistent motion patterns, and to detect respective anomalies simultaneously. In the second branch, we leverage optical flow magnitude estimation for motion anomaly detection and we propose an attention mechanism to engage context information in normal motion modeling and to detect motion anomalies with attention to object parts, the direction of motion, and the distance of the objects from the camera. Our qualitative results show that the proposed method considers the object class effectively and learns motion with attention to the aforementioned determinant factors which results in precise motion modeling and better motion anomaly detection. Additionally, quantitative results show the superiority of our method compared with state-of-the-art methods.