BAT: Behavior-Aware Temporal Contrastive Video Representation Learning

Considering motion behavior in endoscopy, we introduce a pioneering approach named Behavior-Aware Temporal Contrastive Video Representation Learning, referred to as BAT. This is the first method that doesn’t rely on random assembly of clips to create positive and negative pairs for intra-inter contrastive learning. Notably, it is also the first method to establish inter-positive pairs among different videos. The validation of our approach included diverse assessments. Initially, we gathered 239 patient videos recorded in Flexible Endoscopic Evaluation of Swallowing (FEES) to assess penetration/aspiration detection during swallowing. Additionally, we evaluated polyp resection/clipping detection using the HyperKvasir endoscopy dataset. Tasks encompassed both fully supervised and semi-supervised categories across both datasets. BAT achieved accuracy of 72.8%/78.9% for fully supervised/semi-supervised FEES videos (79.9%/91.3% for HyperKvasir). BAT surpassed the state-of-the-art endoscopic video representation learning methodologies by significant margins of 29.1% and 24.6% for FEES, and 6.0% and 12.3% for HyperKvasir, respectively.