Application of Semantic Image Segmentation for Efficient UAS-SfM Photogrammetry Mapping

Structure from Motion (SfM) photogrammetry in combination with the Multi-View Stereo (MVS) technique (SfM-MVS ) is a low-cost and effective tool to reconstruct the 3D structure of the real-world environments and objects using a set of overlapping images. This is approach is often implemented using imagery collected by a small uncrewed aircraft system (UAS) equipped with a high-resolution digital camera (referred to as UAS-SfM). The generated dense point cloud is typically considered the ultimate product of the workflow in almost all commercial and open-source software packages. Although, currently, UAS photogrammetry is being widely used in many remote sensing (RS) applications, there are some challenges regarding the efficiency, quality, and computational complexity in deriving some geospatial products in certain environments, especially where frequent mapping of the area is required to monitor certain changes in the surveyed environment. Efficient mapping of the coastal environments can be challenging due to the presence of water or other moving objects in almost all UAS images. It may also be challenging due to the need for accurate classified 2D and 3D land cover maps or digital terrain models (DTMs) which are required to examine the rate of changes in those environments. Providing those products through the SfM-MVS photogrammetry requires processing a large number of UAS images as well as performing expensive and complicated processing on 2D images and/or 3D point clouds which leads to an exponential increase in the computational complexity and cost in coastal mapping. This study proposes a novel approach that integrates semantic UAS image segmentation into the SfM-MVS photogrammetry workflow to address the aforementioned challenges in UAS photogrammetry mapping in coastal environments. The proposed approach leads to a higher level of automation in generating geospatial products with efficient exploitation of the available computation resources for SfM-MVS computations. The proposed approach exploits state-of-the-art deep learning (DL) for semantic UAS image segmentation to predict land cover image masks. Image masks predicted for moving objects, present in the study area, e.g., water bodies, are used to automatically exclude those areas from SfM-MVS computation. That leads to a higher computation efficiency and an lower noise in the reconstructed model. The proposed approach is then expanded to predict image mask through the DL-based image segmentation model for an automatic generation of the classified point cloud simultaneous with the SfM-MVS processing for reconstruction.