Dual-branch Brain Tumor Classification Based on CNN and Transformer

Brain tumors can disrupt the normal functioning of the nervous system, affect brain functions, and may even lead to the patient’s death. Conventional methods for brain tumors are insufficient to achieve accurate identification of tumor types. In our study, we propose a dual-branch residual transformer fusion network (DRTFNet). The network combines a convolution neural network and a Transformer structure. Within the CNN branch, local information features are extracted using residual dual-shuffled attention modules, and spatial relationships are coupled using a residual-connected channel prior to convolutional attention. Meanwhile, the Transformer branch employs the ResNet structure and Swin Transformer to extract global information characteristics. To reduce redundant information in the feature maps and focus on the key feature information, the ResNet structure incorporates a spatial and channel reconstruction convolution block. Additionally, the Swin Transformer module also utilizes a hierarchical structure and a self-attention mechanism to efficiently process datasets and improve the classification accuracy of the model. We conducted a comprehensive study on multislice brain tumor classification using the brain tumor datasets. Our test findings demonstrate that the proposed deep-learning method can significantly improve classification accuracy while confirming the robustness of our model.