Polyp Tracking in Video Colonoscopy Using Optical Flow With an On-The-Fly Trained CNN

Colonoscopy has been widely applied as a common practice to inspect the inside of large bowel for colon cancer screening. However, missing polyps in such procedure could happen and thus preventing early disease detection and treatment. In this paper, we propose an algorithm for automatic polyp detection and localization in colonoscopy video. The method initially detects and localizes polyps based on single frame object detection or segmentation network such as U-Net. Then it utilizes optical flow to track polyps and fuse temporal information. To overcome tracking failure caused by motion effects, a motion regression model and an efficient on-the-fly trained CNN have been deployed. The proposed algorithm achieves the highest scores in both polyp detection task and polyp localization task in the MICCAI 2018 Endoscopic Vision Challenge on “Gastrointestinal Image Analysis”.