CondDG: Enhancing Autonomous Navigation in Adverse Weather with Conditional Diffusion Synthetic Data

Advancements in autonomous vehicle research have significantly enhanced the performance of object detection and motion planning models, bringing us closer to achieving fully autonomous vehicles. These improvements are largely driven by extensive datasets and innovative model architectures. However, these models continue to struggle with edge-case conditions such as storms, snow, or fog. Addressing these challenges requires di-versified training data that encompasses a wide range of weather conditions. Unfortunately, current methods rely on the manual curation of such datasets, which is both costly and labor-intensive. To tackle this issue, we introduce CondDG (Conditional-Diffusion Data Generation), a novel synthetic data generation technique that leverages the latest advancements in conditional diffusion models to augment datasets with diverse and extreme weather conditions. Our system employs instruction-tuned latent diffusion models and high-quality GPT-40-mini crafted text prompts to generate hyperrealistic synthetic images based on existing annotations, ensuring that the generated data maintains the original image structure. Importantly, our technique operates without the need for human input and enhances bounding box detection in adverse weather conditions. We demonstrate the effectiveness of our approach by balancing data from the BDD100K dataset. Additionally, we show that datasets balanced using our technique outperform unbalanced datasets in terms of detection performance on YOLOv8 by 3–11% across different weather conditions and improve average model confidence by over 13%.