MM-DCNet: A Multi-Scale and Multi-MOdality Dynamic Convolutional Network for Lung Cancer Subtypes Classification

Early and accurate subtypes classifications for lung cancer patients is critical for the following-up treatments and prognosis. CT scan, being the non-invasive and fast-imaging modality, is one of the most common used techniques for cancer diagnosis, thus, the CT-based automatic analysis systems are in high demand. However, the accuracy of such models are limited due to the relative low resolution of CT images. Clinically, pathological examination is regarded as "gold standard" in cancer diagnosis, so the introduction of such cellular-level information into CT-based model is expected to improve model’s accuracy. However, the invasive pathological examination may not be applicable to all clinical scenarios, leading to the presence of numerous unbalanced multi-modality images, i.e., paired CT/pathology images as well as standalone CT images. In this work, we propose a novel classification model, i.e., multi-scale and multi-modality dynamic convolutional network (MM-DCNet), to assist the diagnosis of lung cancer subtypes using unbalanced CT and pathological images. Within the model, we designed a dynamic convolutional module that empowers the model to adaptively adjust its parameters according to different inputs, e.g., the paired multi-modality images and the single-modality image, and consequently exploit the value of all clinical datasets. Furthermore, we designed a contrastive learning module to acquire the cross-modality correlations from paired CT/pathological images and subsequently leverage such correlations as priors to lead the model to more accurate predictions even in the absence of pathology. Experiment results have demonstrated the superiority of our proposed model in lung cancer subtypes diagnosis.