DNAT: Multi-scale Transformer with Dilated Neighborhood Attention for Image Classification

Recently, Transformer-based models have achieved remarkable progress for image classification. In order to reduce the computational costs of multi-head self-attention (MHSA) that is widely adopted in recently-proposed visual Transformers, some window-based Transformers employ local self-attention via shrinking number of input tokens. Nevertheless, these approaches to some extent are short to model long-range dependencies. To address this problem, this paper presents an effective and powerful Transformer backbone, called Dilated Neighborhood Attention Transformer (DNAT) that has a mutli-path architecture, for image classification. DNAT is able to explore long-range interactions through dilated neighborhood attention (DNA). When dilation rate is small, DNAT degenerates to local self-attention that is computationally efficient to capture local features. As the dilation rate increases, more and more long-range token interactions are explored step-by-step. Finally, by integrating weighted features derived from various dilation rates, DNAT seamlessly harvests local-to-global context, yet with acceptable growth of model size and computational budgets. The extensive experiments on ImageNet-1K and CIFAR100 datasets demonstrate the effectiveness of DNAT for image classification.