Toward a Multimodal Deep Learning Approach for Histological Subtype Classification in NSCLC

Accurate classification of non-small cell lung cancer (NSCLC) subtypes is crucial for implementing effective, personalized treatment strategies. This study introduces a novel 3D multimodal convolutional neural network (CNN) architecture for histological subtype classification of NSCLC, specifically distinguishing between lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC). In the context of multimodal deep learning, our approach employs an intermediate fusion technique to integrate both PET and CT imaging modalities, leveraging the complementary information provided by each. We utilized two public datasets alongside a private one, encompassing a total of 714 patients. Our multimodal method was compared against unimodal methods using either CT or PET images alone, achieving better performance. Interestingly, we found that integrating information from multiple imaging modalities can lead to more accurate and reliable NSCLC subtype classification also in case of skewed a priori sample distributions. This non-invasive method has the potential to enhance diagnostic accuracy, improve treatment decisions, and contribute to more personalized and effective lung cancer care strategies. The source code for the implementation described in this paper is available at https://github.com/aksufatih/multimodal-histology-classification