Evaluation Tool to Diagnose Faults and Discrepancy in Semi-Automated or Manual Annotations in Ultrasound Cine Loops (Videos)

Good quality (annotated) data is one of the most important aspects of supervised deep learning. Tasks such as semantic segmentation have a huge data requirement in exchange for only satisfactory performance. Large-scale annotations spread across multiple annotators tends to create inconsistencies, as there are various manual and semi-automated techniques involved. This mandates an external evaluator or expert to check and narrow down the problematic annotations. Studies have shown that even marking a few instances wrong in classification can lead to a significant performance drop in the model (mislabeling only 10% of one class can degrade the total performance of all classes by up to 10%). It has been noticed that fault localization by a medical expert is one of the most expensive and time-consuming processes. In this paper, we propose a novel framework for detecting the inconsistencies in the annotation of every object/anatomy in a specific image. We leverage the power of semi-supervised deep learning models (STCN) to help produce high-quality data for AI segmentation algorithms. Evaluation using this algorithm has been shown to reduce annotation review time by at least 5 hours for just 1000 images, and the quality of ground truth data improved thereby increasing the performance of the model by almost 3%.