Hybrid Vision Transformer for Classification of Pancreatic Cystic Lesions on Confocal Laser Endomicroscopy Videos

The early detection of pancreatic cystic lesions plays a significant role in the survival chance of a patient with pancreatic cancer. Yet it is still a huge challenge. Unfortunately, most pancreatic cancers were diagnosed when the tumor was metastatic. In this study, the Hybrid Transformer, which is the combination of the VGG19 network and Vision Transformer, is utilized as a learning model to predict the pancreatic cystic symptom types in needle-based confocal laser endomicroscopy. A total of 16,944 images containing five types of pancreatic cystic are collected as the training and validation data. Our method can automatically classify the feature type of pancreatic cystic in the test videos and record the prediction results frame by frame. In our experiment, the proposed method successfully identifies the symptom types of 13 from 18 test videos and achieves an accuracy as high as 72%.