Breast Cancer Image Classification: Leveraging Deep Learning and Large Language Models for Semantic Integration

Breast cancer, a significant global health problem that is mainly affecting women, demonstrates the importance of early detection to enhance survival rates. Medical image classification is a significant field that utilizes deep learning methods such as Convolutional Neural Networks (CNNs), Region-Based CNNs (R-CNNs), and Deep Neural Networks (DNNs). Despite this, challenges such as over fitting persist, particularly with limited data or complex models. Although these models learn hierarchical features, interpreting them in intricate image structures is still challenging and complicates decision rationale. To address this, we propose a comprehensive approach for breast cancer image classification. The initial process involves image segmentation using CNN to identify regions of interest, followed by precise object detection with OpenCV. Data quality and diversity are ensured by preprocessing and augmentation with ImageDataGenerator, while informative features are extracted using the InceptionV3model. Our analysis of textual descriptions and embeddings is done by integrating Sentence Transformers and CLIP from the Large Language Model (LLM) framework. Achieving robust decision-making is ensured by setting a threshold of 0.7 for image classification based on predicted malignancy probability. Finally, improving the computations of similarity scores can improve prediction accuracy for reliable diagnostic interpretations. An accuracy of 0.92 in breast cancer diagnosis is guaranteed by integrating LLM and CNN for classification.