Enhanced Breast Cancer Diagnosis: Leveraging Customized Transfer Learning with Machine Learning and Attention Mechanisms for Histopathology Image Classification

Breast cancer diagnosis relies heavily on accurate histopathological image analysis, necessitating advanced methodologies. This study presents a comprehensive deep learning framework integrating preprocessing, feature extraction via customized VGG16 transfer learning model, hybridization with XGBoost, and incorporating attention mechanisms such as self-attention and convolutional block attention module (CBAM). The research aims to develop and assess deep learning models for precisely categorizing benign and malignant samples across various magnifications using the BreaKHis dataset, comprising of breast tumor histopathology images. Various preprocessing techniques, including color normalization, data augmentation, and Synthetic Minority Over-sampling Technique (SMOTE), are deployed to address challenges like class imbalance and enhance model robustness. Results exhibit significant enhancements in classification performance and interpretability, with accuracy levels ranging from 90% to 99% across all magnifications. VGG16, incorporating a convolutional block attention module, demonstrates exceptional performance, underscoring substantial advancements in breast cancer histopathology image classification and model interpretability. These findings position attention mechanisms like CBAM as critical contributors to enhanced diagnostic accuracy and performance.