Towards a Benchmark EO Semantic Segmentation Dataset for Uncertainty Quantification

In order to achieve the objective of accurate and reliable use of deep neural networks for Earth Observation in large-scale scene understanding and interpretation, a large and diverse dataset with proper quantification of uncertainty is required. In this work, we exemplify the lack of a benchmark dataset and present the progress of a novel benchmark dataset for uncertainty quantification of deep learning models in the classic problem of building segmentation from overhead imagery. We present a synthetic dataset where synthetic UAV images were rendered from 3D mesh models of Berlin, Germany. The building masks were extracted from precise LoD-2 building models of the same area. We compare and contrast the performances of baseline methods for semantic segmentation and various uncertainty quantification techniques on this dataset. The experiments show that U-Net is the most accurate model with mIoU of 0.812. Moreover, the Bayesian model is found to be the most reliable uncertainty quantification method on our dataset, with the least ECE.