Different Approaches for Automatic Nucleus Image Segmentation in Fluorescent in situ Hybridization (FISH) Analysis for HER2 Status Assesment

According to American Cancer Society breast cancer is the most common cancer type in women. For most effective treatment choice and patients' state of health prediction it is necessary to make a differential diagnosis to determine breast cancer subtype. The tumor subtype is determined by immunohistochemical or immunocytochemical studies, which evaluate the expression levels of steroid hormone receptors, proliferative protein Ki -67, and oncoprotein CerbB-2 (HER2/neu). HER2-positive subtypes are most adverse (about 25-30% of all cases). In case of indefinite CerbB-2 expression fluorescence in situ hybridization (FISH) investigation is utilized. In most cases, this study is held by visual estimation of fluorescent image parameters by pathologist and thus is subjective. We need to employ automatization techniques to decrease human factor impact and increase reproducibility of the analysis result. FISH analysis automatization for HER2 amplification can be divided into three tasks: nucleus segmentation, signal detection and presentation of the results according to ASCO/CAP recommendations. In this article results for nucleus segmentation task using different machine learning algorithms are presented. The image database for investigations consisted of RGB fluorescent images, as well as gray scale images for each individual fluorophore. The best result was achieved using the random forest algorithm on gray-scale images of individual fluorophores.