Clumped Nuclei Segmentation with Adjacent Point Match and Local Shape-Based Intensity Analysis in Fluorescence Microscopy Images

Highly clumped nuclei captured in fluorescence microscopy images are commonly observed in a wide spectrum of tissue-related biomedical investigations. To ensure the quality of downstream biomedical analyses, it is essential to accurately segment clustered nuclei. However, this presents a technical challenge as fluorescence intensity alone is often insufficient for recovering the true nuclei boundaries. In this paper, we propose an segmentation algorithm that identifies point pair connection candidates and evaluates adjacent point connections with a formulated ellipse fitting quality indicator. After connection relationships are determined, we recover the resulting dividing paths by following points with specific eigenvalues from the image Hessian in a constrained searching space. We validate our algorithm with 560 image patches from two classes of tumor regions of seven brain tumor patients. Both qualitative and quantitative experimental results suggest that our algorithm is promising for dividing overlapped nuclei in fluorescence microscopy images widely used in various biomedical research.