CrackMDM: Masked Modeling on DCT Domain for Efficient Pavement Crack Segmentation

Existing pavement crack inspection methods heavily rely on a large amount of annotated samples and require overburdened computational power, which is not affordable on edge devices. Recent methods mostly focus on spatial features, which cannot effectively capture the long, continuous, tender, and thin road features. The masked image modeling (MIM) is an effective way to rebuild the crack primitive features by masking strategy on unlabeled data to reduce the dependence on annotated data, and the convolution on the frequency domain provides a flexible and efficient path to capture continuous and tender road features. Inspired by the thoughts of masked frequency modeling (MFM), we proposed a masked discrete cosine transform (DCT)-domain modeling strategy, named crack masked DCT-domain modeling (CrackMDM), for efficient pavement crack segmentation. Specifically, we propose a DCT-domain masked modeling method in the CrackMDM model, which combines the advantages of separable convolutions and spectral convolutions (SP-Convs) in the DCT domain to extract continuous and tender crack structures. Additionally, we introduce the self-supervised pretraining with a masking strategy in the DCT domain using unlabeled crack samples to build crack primitives and to fine-tune the encoder and decoder parameters on labeled crack data to refine crack features in the fine-tuning phase. The CrackMDM model is evaluated on three public benchmarks: CFD, YCD, and GAPs, and achieves state-of-the-art (SOTA) performance with superior inference speed. Codes are available at https://github.com/Jyuan357/CrackMDM