Methods for Measuring Geodiversity in Large Overhead Imagery Datasets

Geographic variation in the appearance of objects on Earth is readily observable in remotely sensed imagery (RSI) and somewhat intuitive to understand for most people - many classes of objects (houses, vehicles, crop fields etc.) simply look different depending on their location. This variation has recently been shown to have important implications when training machine learning models on geotagged image datasets for specific object detection and classification tasks. For example, models trained on datasets with ethnocentric biases in image content have been shown to misclassify objects in under-sampled regions, particularly in least-developed countries. The need to evaluate the growing corpus of RSI datasets for representativeness, heterogeneity and geodiversity is therefore high; yet scalable methods for measuring these concepts are absent in the remote sensing domain. This paper introduces the first dataset analysis methods for detecting and assessing geodiversity problems in large RSI datasets, based on geospatial adaptations of the Fréchet Inception Distance and Inception Score in the deep learning framework. Geospatial Fréchet Distance is proposed as a dissimilarity measure for image features of an object class across geographic regions - useful for comparing differences in object class appearance in different locations and/or spatial scales. A complementary Geospatial Inception Score is proposed to quantify heterogeneity of geographic context present in dataset labels within particular regions/locations, taking into account the labels themselves as well as their immediate surroundings. Rigorous tests of these methods on simulated RSI datasets demonstrate their stability, sensitivity, and the broad range of dataset analyses to which they can be applied.