Multi-Modal Vision Transformer using Variable and Shifted Windows for MRI Reconstruction

Swin Transformer has become a paradigm for high-level vision tasks. However, its application in image reconstruction is hampered by the fixed window size and high-resolution feature maps, which restrict the extent of information extraction. In this work, we propose a novel multi-modal vision Transformer network using variable and shifted windows (MVSWT) to tackle the aforementioned challenges for accelerated MRI reconstruction. The variable and shifted windowing strategy achieves flexible local and long-range dependency modelling by focusing on multi-scope self-attention calculations. To fully leverage the multi-modality advantage of MRI, we construct a cross-modal information interaction architecture that incorporates the cross attention mechanism. Moreover, the introduction of data consistency (DC) operation enables the Transformer-based network to learn and estimate residual images for fast and accurate convergence. Qualitative and quantitative experimental results demonstrate that MVSWT reaches more competitive performance against other benchmark algorithms.