Towards a Multi-Agent Vision-Language System for Zero-Shot Novel Hazardous Object Detection for Autonomous Driving Safety

Detecting anomalous hazards in visual data, particularly in video streams, is a critical challenge in autonomous driving. Existing models often struggle with unpredictable, out-of-label hazards due to their reliance on predefined object categories. In this paper, we propose a multimodal approach that integrates vision-language reasoning with zero-shot object detection to improve hazard identification within traffic scene data. Our approach consists of a pipeline that integrates both Vision-Language Model (VLM) and Large-Language Model (LLM) capabilities to solve the problem of detecting out-of-label hazards within traffic scene data. We refine object detection by incorporating OpenAI’s CLIP model to match predicted hazards with bounding box annotations, improving localization accuracy. To assess model performance, we extended the foundational COOOL (Challenge-of-Out-of-Label) anomaly detection benchmark dataset by adding full natural language hazard descriptions as ground truth annotations. Additionally, we evaluated our pipeline’s hazard labels through cosine similarity in order to consider the semantic similarity between the predicted hazard description and the annotated ground truth for each video. Our findings highlight the strengths and limitations of current vision-language based approaches, offering insights into future improvements in autonomous hazard detection systems. Our models, scripts, and data can be found at https://github.com/mi3labucm/COOOLER.git