A CNN-Transformer combined Remote Sensing Imagery Spatiotemporal Fusion Model

Remote sensing images (RSIs) Spatiotemporal fusion (STF) make a significant contribution to acquisition RSIs sequence with simultaneously high temporal and spatial resolution, which broadens its application fields. However, the existing RSIs STF methods lack effective strategies for extracting global information and fusion features between different images. Conversely, the existing State-Of-The-Art (SOTA) methods generally require more than two RSIs from different satellites as reference, which increases the difficulty of data collection and limits the application in practice. To address these problems, this paper proposed an end-to-end CNN and Transformer combined RSIs STF model (CTSTFM) based on two reference RSIs. Specifically, the proposed CTSTFM consists of three basic modules: Multi Kernel Convolutional Transformer encoder (MKCE), Cross Fusion module (CFM) and Convolutional-based Compression decoder (CCD). The MKCE combines Multi Kernel Channel Attention block and Multi Kernel Spatial Attention block to extract shadow features and long-term interdependencies in reference RSIs. The CFM uses the unique Cross Exchange Transformer block and Combine Fusion Transformer block to enhance the feature fusion results. Due to the powerful Encoder and Fusion Module, in the CCD part we only use a simple design convolution module to save the consumption of computational resources. Experiments on two well-known open access datasets shows that CTSTFM achieves competitive results in both qualitative and quantitative comparisons compared to the SOTA methods. Meanwhile, we conduct experiment to analyze the image Tessellation Effects and its solution. The effectiveness of the proposed module will be demonstrated through ablation experiments