A Mumford-Shah functional based variational model with contour, shape, and probability prior information for prostate segmentation

Inter patient shape, size and intensity variations of the prostate in transrectal ultrasound (TRUS) images challenge automatic segmentation of the prostate. In this paper we propose a variational model driven by Mumford-Shah (MS) functional for segmenting the prostate. Parametric representation of the implicit curve is derived from principal component analysis (PCA) of the signed distance representation of the labeled training data to impose shape prior. Posterior probability of the prostate region determined from random forest classification facilitates initialization and propagation of our model in a MS energy minimization framework. The proposed method achieves mean Dice similarity coefficient (DSC) value of 0.97±0.01, with a mean Hausdorff distance (HD) value of 1.73±0.24 mm when validated with 24 images from 6 datasets in a leave-one-patient-out validation framework. The model achieves statistically significant t-test p-value<0.0001 in mean DSC and mean HD values compared to traditional statistical models of shape and appearance.