Optimizing and Evaluating Swin Transformer for Aircraft Classification: Analysis and Generalizability of the MTARSI Dataset

Aircraft classification via remote sensing images has many commercial and military applications. The Swin-Transformer has shown great promise, recently dominating general-purpose image classification benchmarks such as ImageNet. In this paper, we test whether the performance of the Swin-Transformer on general-purpose image classification translates to domain-specific aircraft classification using the Multi-Type Aircraft from the Remote Sensing Images dataset. We also investigate the effect of training procedure vs. model selection on the validation score. Our carefully trained Swin-Transformer model achieved an impressive 99.4 % validation set accuracy without super-resolution, and 99.5 % with super-resolution. Moreover, the generalization of models trained on the MTARSI dataset to real-world and synthetic aircraft classification is evaluated with some out-of-distribution samples. Our results demonstrate that the lack of complexity and heterogeneity of the MTARSI dataset, and the labeling errors resulted in models which struggle to achieve high accuracy on the adopted test samples despite near perfect validation scores.