Pyramid Transformer Driven Multibranch Fusion for Polyp Segmentation in Colonoscopic Video Images

Colonoscopic polyp segmentation is essential and valuable to early diagnosis and treatment of colorectal cancer. It remains challenging to accurately extract these polyps due to their small sizes, irregular shapes, image artifacts, and illumination variations. This work proposes a new encoder-decoder architecture called pyramid transformer driven multibranch fusion to precisely segment different types of colorectal polyps during colonoscopy. Specifically, our architecture employs a simple, convolution-free pyramid transformer as its encoder that is a flexible and powerful feature extractor. Next, a multibranch fusion decoder is employed to reserve the detailed appearance information and fuse semantic global cues, which can deal with blurred polyp edges caused by nonuniform illumination and the shaky colonoscope. Additionally, a hybrid spatial-frequency loss function is introduced for accurate training. We evaluate our proposed architecture on colonoscopic polyp images with four types of polyps with different pathological features, with the experimental results showing that our architecture significantly outperforms other deep learning models. Particularly, our method improves the average dice similarity and intersection over union to 90.7% and 0.848, respectively.