Multi-Modal Diffusion for Self-Supervised Pretraining

Self-supervised pretraining has been shown to greatly improve the performance and label-efficiency of Deep Learning models for multi-modal remote sensing applications. In this work, we explore the use of diffusion methods in a multi-modal setup as a means to pretrain architectures in a task-agnostic way. We qualitatively find that multi-modal diffusion processes are able to coherently learn information across multiple data-modalities. Fine-tuning the pretrained backbone on a segmentation task outperforms a supervised baseline model and is highly label-efficient, which is not the case when fine-tuned on a classification task.