Apply Masked-attention Mask Transformer to Instance Segmentation in Pathology Images

Instance segmentation can be applied for the discrimination and diagnosis of cancer cells in pathology images. Accurate segmentation of each pathological cell in the pathology images can improve the efficiency of clinical diagnosis. In this paper, we aim to evaluate the state-of-the-art transformer-based instance segmentation method, masked-attention mask transformer (Mask2Former)[1], on pathology datasets. With the pretrained model of Mask2Former on the natural image instance segmentation dataset, we show that Mask2Former can be adaptive to small pathological datasets and achieve comparable or even better instance segmentation performance compared with the state-of-the-art task-specific pathology image instance segmentation methods.

I. Introduction

The identification of tumor tissues from pathology images provides an early evidence of cancer diagnosis. Pathologists can use the pathology images to determine whether the tissue is benign or malignant, as well as the degree of inflammation. Furthermore, the area of malignant regions in the images can be identified, and the border of the malignant regions can be drawn to show the degree of the disease. Nevertheless, the manual process is time-consuming and subjective. Thus, artificial intelligence techniques are expected to assist pathologists for disease diagnosis from pathology images [2].