HazardVLM: A Video Language Model for Real-Time Hazard Description in Automated Driving Systems

Early and accurate hazard detection is crucial for enhancing the safety of automated driving systems (ADS) by allowing them to anticipate and react to imminent accidents. While current research focuses on classifying safe and hazardous scenes, simply identifying a hazard class does not provide sufficient information for an appropriate response. This gap underscores the need for an effective language model to describe hazards and enable downstream ADS planning of evasive manoeuvres. Such a model could also be integrated into a warning system for human-driven vehicles and assist developers and regulators in interpreting system responses. Motivated by this need, we propose HazardVLM, a novel video language model designed for real-time hazard description. Conventional approaches leverage generic large language models but are computationally expensive and known to generate inconsistent and inaccurate information in specific tasks. To this end, HazardVLM is a task-specific small language model which reduces complexity through 1) a lightweight encoder-decoder architecture, 2) a custom tokenizer with only relevant vocabulary to create a task-specific problem space and 3) adaptive sampling using optical flow to reduce the complexity of video analysis. To evaluate HazardVLM, we introduce a new dataset of over 7,901 real-world road collision scenes with detailed descriptions. Our extensive evaluation demonstrates that HazardVLM outperforms state-of-the-art models (SOTA) by 9.59% in hazard description accuracy, achieving a BLEU-4 score of 0.714 and reducing the critical false negative rate (missed hazards) by 26.9% compared to the SOTA.