Lightweight and Efficient: A Family of Multimodal Earth Observation Foundation Models

This paper explores novel developments in Earth Observation (EO) foundation models, with a focus on leveraging variants of the Vector-Quantized Variational Autoencoders (VQ-VAE). We build on the CORSA architecture [1], [2] designed for onboard compression but extend this to a suite of models with each their own strengths. We explore this family of lightweight models by adding a vision transformer backbone and comparing this to convolutional variants. We also add a masked image modelling pretraining objective next to the original reconstruction objective to improve the quality of the representations. Finally, we extend this suite of models to a multi-modal setup that can handle Sentinel-2 as well as Sentinel-1 inputs. To analyse the effect of all these design choices, we evaluate the models on the downstream tasks of landcover classification and change detection.