The Validation Index: A New Metric for Validation of Segmentation Algorithms Using Two or More Expert Outlines With Application to Radiotherapy Planning

Validation is required to ensure automated segmentation algorithms are suitable for radiotherapy target definition. In the absence of true segmentation, algorithmic segmentation is validated against expert outlining of the region of interest. Multiple experts are used to overcome inter-expert variability. Several approaches have been studied in the literature, but the most appropriate approach to combine the information from multiple expert outlines, to give a single metric for validation, is unclear. None consider a metric that can be tailored to case-specific requirements in radiotherapy planning. Validation index (VI), a new validation metric which uses experts' level of agreement was developed. A control parameter was introduced for the validation of segmentations required for different radiotherapy scenarios: for targets close to organs-at-risk and for difficult to discern targets, where large variation between experts is expected. VI was evaluated using two simulated idealized cases and data from two clinical studies. VI was compared with the commonly used Dice similarity coefficient
${\rm (DSC}_{{\rm pair} - {\rm wise}})$
and found to be more sensitive than the
${\rm DSC}_{{\rm pair} - {\rm wise}}$
to the changes in agreement between experts. VI was shown to be adaptable to specific radiotherapy planning scenarios.