PointFormer: Keypoint-Guided Transformer for Simultaneous Nuclei Segmentation and Classification in Multi-Tissue Histology Images

Automatic nuclei segmentation and classification (NSC) is a fundamental prerequisite in digital pathology analysis as it enables the quantification of biomarkers and histopathological features for precision medicine. Nuclei appear to be small, however, global spatial distribution and brightness contrast, or color correlation between the nucleus and background, have been recognized as key rationales for accurate nuclei segmentation in actual clinical practice. Although recent great breakthroughs in medical image segmentation have been achieved by Transformer-based methods, the adaptability of segmenting and classifying nuclei from histopathological images is rarely investigated. Also, the severe overlap of nuclei and the large intra-class variability are common in clinical wild data. Prevailing methods based on polygonal representations or distance maps are limited by empirically designed post-processing strategies, resulting in ineffective segmentation of large irregular nuclei instances. To address these challenges, we propose a keypoint-guided tri-decoder Transformer (PointFormer) for NSC simultaneously. Specifically, the overall NSC task is decoupled to a multi-task learning problem, where a tri-decoder structure is employed for decoding nuclei instance, edges, and types, respectively. The nuclei detection and classification (NDC) subtask is reformulated as a semantic keypoint estimation problem. Meanwhile, introduces a novel attention-guiding strategy to capture strong inter-branch correlations and mitigate inconsistencies between multi-decoder predictions. Finally, a multi-local perception module is designed as the base building block of PointFormer to achieve local and global trade-offs and reduce model complexity. Comprehensive quantitative and qualitative experimental results on three datasets of different volumes have demonstrated the superiority of the proposed method over prevalent methods, especially for the PanNuke dataset with an achievement of 70.6% on bPQ.