Deep Learning-Based Spatiotemporal Fusion of Unmanned Aerial Vehicle and Satellite Reflectance Images for Crop Monitoring

Spatiotemporal fusion (STF) techniques play important roles in Earth observation analysis as they enable the generation of images with high spatial and temporal resolution. However, existing STF models often fuse images from various satellites, not satisfying the demand for precise crop monitoring. In contrast, unmanned aerial vehicle (UAV) images can deliver detailed data, and deep learning (DL)-based STF models have the potential to automatically extract abstract features. To this end, this study proposed a novel end-to-end DL-based STF model named UAV-Net, which can produce centimeter-scale UAV images. UAV-Net has an encoder-decoder architecture with Modified ResNet (MResNet), Feature Pyramid Network (FPN), and decoder modules. The encoder uses MResNet modules to extract input features, while the FPN module performs a multiscale fusion of these features before reconstructing UAV images using transposed convolution in the decoder module. Through comparative and ablation experiments, this study evaluated the efficacies of MResNet modules with 18, 34, and 50 layers, along with the FPN module of UAV-Net. The experimental results on real-world datasets demonstrated that UAV-Net adequately produces UAV images both visually and quantitatively. Furthermore, a comparison with state-of-the-art STF models highlights the innovation and effectiveness of UAV-Net in producing centimeter-scale images. The predicted centimeter-scale images using UAV-Net have great potential for various environmental monitoring applications.