An Active and Contrastive Learning Framework for Fine-Grained Off-Road Semantic Segmentation

Off-road semantic segmentation with fine-grained labels is necessary for autonomous vehicles to understand driving scenes, as the coarse-grained road detection cannot satisfy off-road vehicles with various mechanical properties. Pixel-wise annotation of fine-grained labels in off-road scenes is very hard because a large part of the pixels could suffer from severe semantic ambiguity. Furthermore, semantic properties of off-road scenes can be very changeable due to various precipitations, temperature, defoliation, etc. To address these challenges, this research proposes an active and contrastive learning-based method. A few image patches are annotated mainly to distinguish semantic differences rather than semantic categories, which can greatly reduce the burden of manual annotation. A feature representation is learnt using the contrastive pairs of image patches, and semantic categories are adaptively modeled from the data. To actively adapt to new scenes, a risk evaluation method is developed to discover and select hard frames with high-risk predictions for supplementary labeling, to update the model efficiently. Extensive experiments and analyses are conducted on self-developed and public datasets. Experimental results demonstrate that fine-grained semantic segmentation can be learned with only dozens of weakly labeled frames, and the model can efficiently adapt across scenes by weak supervision, while achieving competitive performance with the typical fully supervised ones.