MarsScapes and UDAFormer: A Panorama Dataset and a Transformer-Based Unsupervised Domain Adaptation Framework for Martian Terrain Segmentation

Martian terrain segmentation aims to assign all pixels of an input image with various terrain labels, which provides a firm support for the downstream research on rover traversing and geologic analysis tasks. However, existing studies in this field suffer from limitations in two aspects: one is the lack of large-scale and high-quality Martian terrain datasets, and the other is the over-reliance on purely supervised learning that is very data-hungry and sensitive to domain shifts among different datasets. In this article, we overcome these from the perspective of both data and methodology. First, we publish MarsScapes, a panorama dataset with appreciable data volume and fine-grained annotations for Martian terrain understanding. The dataset contains 195 terrain panoramas composed of 3779 subimages, and all pixels in the panoramas are split into nine semantic categories. Then, we propose the first transformer-based unsupervised domain adaptation (UDA) framework (UDAFormer) for the cross-domain terrain segmentation on Mars, which consists of a teacher–student model and an output-guided biased sampling (OGBS) module. The teacher–student model performs knowledge distillation to explore robust cross-domain features, where a modified augmentation regularization (MAR) is designed to alleviate the interference of undesirable augmentations to domain adaption. The OGBS helps the teacher–student network to emphasize the categories that tend to be ambiguous or submerged during the training, elevating the overall accuracy for the UDA segmentation of Martian terrains. Extensive experiments on the MarsScapes and another dataset called Mars-Seg demonstrate the superiority of UDAFormer over the state-of-the-art methods in UDA Martian terrain segmentation.