MS-TFGNet: A Dual-Branch Transformer for Guided Learning in Lung Cancer Lesion Segmentation Using PET/CT Images

Lung cancer is the most common and lethal malignant tumor globally, characterized by a tendency for local invasion and distant metastasis, which complicates detection. Thus, accurate lesion segmentation for lung cancer is crucial. Positron emission tomography-computed tomography (PET/CT) provides both anatomical information and metabolic information about tissues in the clinical diagnosis of lung cancer. However, existing PET/CT -based multimodal segmentation strategies typically achieve segmentation by concatenating and fusing the features extracted from both modalities. These approaches often overlook the complementary nature of the feature extraction process between the different modalities. To address these issues, we propose an end-to-end deep learning architecture, called the Multi-scale Transformer Feature Fusion and Guided Network (MS-TFGNet), aimed at improving the accuracy of lung cancer lesion segmentation using PET/CT images. Specifically, a dual-branch structure was designed to integrate multimodal features, the PET provides metabolic information to guide the CT branch using a Multi-modal Cross-Attention Transformer (MCA) module during training. Finally, MS-TFGNet was validated on a real clinical PET/CT dataset, achieving a DSC, precision, and recall of 0.8502, 0.8401, and 0.8610, respectively.