Building Extraction With Vision Transformer

As an important carrier of human productive activities, the extraction of buildings is not only essential for urban dynamic monitoring but also necessary for suburban construction inspection. Nowadays, accurate building extraction from remote sensing images remains a challenge due to the complex background and diverse appearances of buildings. The convolutional neural network (CNN)-based building extraction methods, although increased the accuracy significantly, are criticized for their inability for modeling global dependencies. Thus, this article applies the vision transformer (ViT) for building extraction. However, the actual utilization of the ViT often comes with two limitations. First, the ViT requires more GPU memory and computational costs compared with CNNs. This limitation is further magnified when encountering large-sized inputs like fine-resolution remote sensing images. Second, spatial details are not sufficiently preserved during the feature extraction of the ViT, resulting in the inability for fine-grained building segmentation. To handle these issues, we propose a novel ViT (BuildFormer), with a dual-path structure. Specifically, we design a spatial-detailed context path to encode rich spatial details and a global context path to capture global dependencies. Besides, we design a window-based linear multihead self-attention whose complexity is linear with the window size. Such a design allows the BuildFormer to apply large windows for capturing global context, which greatly improves its potential in processing large-sized remote sensing images. The proposed method yields the state-of-the-art performance (75.74% IoU) on the Massachusetts building dataset. Code will be available at
https://github.com/WangLibo1995/BuildFormer
.