Robust Traffic Sign Detection in Adverse Weather via Hybrid Vision Transformer and GAN-based Augmentation

The performance of traffic sign detection systems, a critical component for autonomous driving, is severely compromised by adverse weather conditions, particularly fog. While Vision Transformers (ViTs) have emerged as a powerful alternative to traditional CNNs, they often exhibit limitations in capturing fine-grained local features and require extensive training data, a challenge compounded by the scarcity of real-world foggy-weather datasets. To address these issues, this paper proposes a novel and robust framework that enhances ViT-based detectors through two key innovations: 1) A lightweight hybrid architecture (FogAug-HyViT) that synergizes the global context modeling of ViTs with the local feature extraction capabilities of CNNs via novel convolutional bridge layers. 2) A targeted data augmentation strategy employing Foggy-CycleGAN to synthesize realistic foggy images, thereby creating a more diverse and robust training corpus. We conduct extensive experiments on the German Traffic Sign Recognition Benchmark (GTSRB). Our results show that GAN-based augmentation significantly boosts performance, with our standard FogAug-ViTBase model’s mAP increasing from 0.851 to a strong 0.887. Notably, our proposed FogAug-HyViT achieves a near-perfect Average Precision (AP) of 0.993 on large objects, validating the effectiveness of the convolutional bridges. These models not only outperform a strong ResNet-50 baseline but also approach the performance of a much larger ViT-Base model, demonstrating the efficacy of our approach in developing robust and data-efficient perception systems for all-weather autonomous driving.