MST-Net: A General Deep Learning Model for Thick Cloud Removal From Optical Images

Temporally neighboring homologous images are crucial to provide auxiliary information for thick cloud removal. Due to the inherent satellite revisit period and frequent cloud obscuration, there is often a significant time interval between the target cloudy images and neighboring cloud-free homologous images, leading to potential land surface condition changes. Moreover, multitemporal cloudy images that may contain valuable complementary information in the noncloudy regions, are often neglected in practice. This article focused on thick cloud removal from Landsat 8 OLI images. We proposed to fuse the temporally more frequent Sentinel-2 MSI images and also cloudy multitemporal images consisting of Sentinel-2 MSI and Landsat 8 OLI time-series. Acquired by a sensor different from Landsat 8 OLI, Sentinel-2 MSI images exhibit great similarities in data characteristics. To fully exploit the spatio-temporal-spectral information in multisource and multitemporal auxiliary images, we proposed a novel deep network called MST-Net. MST-Net was validated using 12 simulated and two real cloudy Landsat 8 OLI images. The results show that the MST-Net can produce more satisfactory predictions than the five benchmark methods. Both the images acquired by a different sensor and homogeneous multitemporal cloudy images are beneficial. Under different sizes of clouds, the MST-Net produces consistently the most accurate predictions. Furthermore, due to the fusion of all bands simultaneously in the temporally closest Sentinel-2 MSI images, the MST-Net is less affected by thin cloud occlusion errors. Overall, the MST-Net shows great potential for cloud removal from optical images produced by a wide range of sensors and, more generally, filling gaps in various global scale products.