Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

In this paper, we address panoramic semantic segmentation which is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of semantic annotations in the
$360^\circ$
imagery. To tackle these problems, first, we propose the upgraded Transformer for Panoramic Semantic Segmentation, ie, Trans4PASS+, equipped with
Deformable Patch Embedding (DPE)
and
Deformable MLP (DMLPv2)
modules for handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels). Second, we enhance the
Mutual Prototypical Adaptation (MPA)
strategy via pseudo-label rectification for unsupervised domain adaptive panoramic segmentation. Third, aside from Pinhole-to-Panoramic (
Pin2Pan
) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images, facilitating Synthetic-to-Real (
Syn2Real
) adaptation scheme in
$360^\circ$
imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with
Pin2Pan
and
Syn2Real
regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at
https://github.com/jamycheung/Trans4PASS
.