Application in Tumor Classification Based on Target Morphological Consistency Patches Adaptively Extracted and Priori Guided Feature Extraction Method

PET/CT images have the characteristics of accurate location and high sensitivity for tumor lesions, so the classification on PET/CT images is of great significance in clinical practice. In terms of visual tasks such as image classification, models such as vision transformer based on attention mechanism has been proved to be effective. However, medical images are different from natural images, and only using image information as model input can not fully and effectively express target characteristics. In this paper, we propose a method of adaptive extraction of the patch to maintain the morphological characteristics of the target, and add the radiomics features to the attention mechanism as a priori guidance knowledge for feature extraction, which is applied to the classification of PET/CT tumors. Firstly, for the original PET/CT image, the patch sequence is obtained by the adaptive patch extraction method based on overlapping region and sliding stride, which adapts to the input of the model while maintaining the morphological characteristics of the target; Secondly, the corresponding image token and radiomics feature vector of patches are obtained through linear projection layer and radiomics feature extractor respectively; Finally, the two are input into the multi-head attention module of the encoder to build a multi-feature attention mechanism, and complete the classification decision after the feature extraction. To verify the effectiveness of this method, we conducted experiments on the whole-body public dataset FDG-PET-CT-Lesions, and classified PET/CT images into negative, melanoma, lymphoma and non-small cell lung cancer, and the classification accuracy reached 0.8652. The experimental results show that the proposed method has a high performance on tumor classification in PET/CT images, and can be applied to classification of various tumors.