MMFSeg: Multi-structure Multi-feature Fusion for Segmentation of Road Potholes

Road pothole segmentation is important for the driving safety of autonomous vehicles, especially in unstructured or rural environments. Recently, many effective multi-modal fusion networks have been proposed for road pothole segmentation. However, most of them adopt two encoders with the same type of structure, such as only convolutional neural network (CNN) or only Transformer, to extract features from different modalities. This overlooks the fact that the information richness of features extracted from different modalities varies across the types of features, such as CNN features or self-attention features. To provide a solution to this issue, we design a novel RGB-Disparity segmentation network, named MMFSeg, by adopting the two types of structures as encoders. Specifically, we adopt CNN and Transformer as encoders to extract two types of features from each modality, and also propose a late-fusion multi-feature alignment fusion module to fuse the two types of features with different numbers of channels. Experimental results demonstrate that our network outperforms well-known networks, and can trade-off between accuracy and efficiency.