FCTformer: Fusing Convolutional Operations and Transformer for 3D Rectal Tumor Segmentation in MR Images

Accurate and reliable segmentation of rectal cancer in Magnetic Resonance Imaging holds crucial significance in preoperative prediction, tumor staging, and neoadjuvant therapy. Currently, the automated segmentation methods for rectal tumor have predominantly relied on Convolutional Neural Networks (CNNs), which lean heavily on discerning the contrast disparities among locally neighboring MRI voxels. However, these methods tend to exhibit segmentation inaccuracies when confronted with rectal cancer instances characterized by indistinct contrasts and markedly diverse shapes. Here, we propose a FCTformer who Fuses Convolutional operations and Transformer modules for accurate rectal tumor segmentation in 3D MRI. Specifically, first, FCTformer integrates a transformer-based global feature extraction mechanism and a CNN-based local feature extraction approach to obtain a dual-faceted multiscale feature representation. This representation enhances the model’s capability to capture both the comprehensive semantic features and intricate details of rectal cancer instances, especially in challenging situations such as low-contrast imaging and substantial shape variations. Second, to capitalize on features captured across different scales, thereby enhancing segmentation accuracy, we have incorporated a Dual-Attention decoder. Third, to enhance the tumor’s edges and contours, the Prediction Aggregation Unit is designed to capture sharper tumor boundaries and retain fine details that could be lost during repetitive down-sampling stages. Experimental results involving 362 instances of rectal tumor segmentation demonstrate that our proposed method achieves a Dice Similarity Coefficient of 0.827, surpassing existing methods. The satisfactory results obtained from evaluating our approach on a publicly available prostate dataset validate its generalizability.