Graph cut energy minimization in a probabilistic learning framework for 3D prostate segmentation in MRI

Variations in inter-patient prostate shape, and size and imaging artifacts in magnetic resonance images (MRI) hinders automatic accurate prostate segmentation. In this paper we propose a graph cut based energy minimization of the posterior probabilities obtained in a supervised learning schema for automatic 3D segmentation of the prostate in MRI. A probabilistic classification of the prostate voxels is achieved with a probabilistic atlas and a random forest based learning framework. The posterior probabilities are combined to obtain the likelihood of a voxel being prostate. Finally, 3D graph cut based energy minimization in the stochastic space provides segmentation of the prostate. The proposed method achieves a mean Dice similarity coefficient (DSC) value of 0.91±0.04 and 95% mean Haus-dorff distance (HD) of 4.69±2.62 voxels when validated with 15 prostate volumes of a public dataset in a leave-one-patient-out validation framework. The model achieves statistically significant t-test p-value<;0.0001 in mean DSC and mean HD values compared to some of the works in literature.