Integrating Segment Anything Model derived boundary prior and high-level semantics for cropland extraction from high-resolution remote sensing images

Visual foundation models (VFMs) pretrained on large-scale training datasets show robust zero-shot adaptability across many vision tasks. However, there still exist limitations in remote sensing processing tasks due to the variety and complexity of remote sensing images. In this letter, we propose a two-flow network based on multi-task VFM, named TFNet, to extract croplands with well-delineated boundaries from high-resolution remote sensing images. TFNet consists of a mask flow and a boundary flow. It first uses a VFM as visual encoder to obtain universal semantic features regarding croplands, and then aggregates them into the two flows. Next, a boundary prior-guided module is developed to incorporate boundary semantics derived from the boundary flow into the mask flow, to refine the boundary details of croplands. We also develop a multi-branch parallel fusion module that aggregates multi-scale contextual information to improve the identification of cropland with varied sizes and shapes. Finally, a semantic consistency loss is introduced to further optimize the feature learning of cropland information. We conducted extensive experiments on Shandong and Xinjiang datasets collected from Gaofen-2 satellites, and compared our method with five existing methods. Experimental results show that the croplands extracted by our method have the fewest omissions and errors, achieving the highest attribute accuracy (IoU of 0.863 and 0.945) and lowest geometric errors (GTC of 0.134 and 0.097) than other methods on the two datasets. Our method effectively distinguished croplands of varied sizes, shapes, and spectra, even in scenarios with limited samples. Code and datasets are available at https://github.com/long123524/TFNet.