Enhancing the Safety of Autonomous Vehicles: Semi-Supervised Anomaly Detection With Overhead Fisheye Perspective

Autonomous vehicles (AVs) have the potential to revolutionize transportation. However, ensuring passenger safety within these vehicles in the absence of a dedicated onboard authority figure necessitates the development of intelligent, autonomous surveillance systems. This paper presents a novel semi-supervised anomaly detection system specifically designed to enhance safety within autonomous shuttles. Our approach leverages overhead fisheye cameras to provide comprehensive, occlusion-resistant monitoring of the cabin interior. This unique perspective maximizes visibility, even in crowded conditions. Our spatiotemporal autoencoder architecture, composed of both convolutional and reccurrent layers, is trained on extensive unlabeled video data to learn representations of regular passenger behavior using a Center-Weighted Loss (CWL) function that focuses in the cabin’s central region, where critical events are most likely to occur. This reduces the potential for false positives triggered by rapid changes on the periphery due to the vehicle’s movement. To enhance the system’s ability to discriminate between specific safety and security incidents, we introduce a classifier fine-tuned on a labeled subset of our dataset. We evaluate our method’s performance through experimentation on a real-world dataset (CERTH-AV) collected with an overhead fisheye camera. Our method demonstrates superior anomaly detection capabilities, achieving the highest Area Under the Curve (AUC) performance on the CERTH-AV dataset. Further comparative evaluations on established benchmarks, including UCF-Crime and ShanghaiTech, validate our system’s robustness and adaptability. Finally, we have successfully integrated our method into autonomous minibuses using NVIDIA Jetson embedded systems for real-time processing, demonstrating the practical efficacy of our approach in safeguarding passengers within autonomous vehicles.