An Optimized Multi-Organ Cancer Cells Segmentation for Histopathological Images Based on CBAM-Residual U-Net

In digital pathology, the accurate segmentation of cell nuclei in histopathology images is essential for medical image analysis. Histopathologists visually evaluate the patterns of cellular architecture and tissue patterns in histopathology image analysis for cancer detection to determine the malignant tissue portions and assess the severity of malignancy. However, manually analyzing scans using a high-resolution microscope requires significant effort and time. A computer-assisted diagnosis system utilizing deep learning (DL) algorithms rapidly, reliably, and automatically segments cell nuclei. However, the existing research studies have limited accuracy, high computational costs, and a lack of robustness and generalizability on diverse datasets. To address these issues, this paper proposes a novel and improved DL architecture based on the U-Net, namely, the CBAM-Residual U-Net for improving accuracy, robustness, and generalized segmentation algorithm that can be applied to various staining techniques and tissue structures. The proposed architecture utilizes a ResConv and convolution block attention modules (CBAM). These modules help the proposed architecture learn the image’s shallow and deep features. The CBAM module uses an attention mechanism concentrating on essential features such as cell nuclei’s shape, texture, and intensity to accurately segment the raw input patterns. The proposed CBAM-Residual U-Net involves fewer trainable parameters, reducing the computational and time cost s compared to state-of-the-art techniques. Extensive experiments and comprehensive evaluations are conducted to demonstrate the performance of the proposed scheme on publicly available datasets: i) Data Science Bowl (DSB) 2018, ii) The GlaS, iii) Triple-Negative Breast Cancer (TNBC). The experimental results show that our proposed model considerably outperforms the state-of-the-art techniques and detects cellular boundaries well, providing fine-grained segmentation results.