Self-distillation Augmented Masked Autoencoders for Histopathological Image Understanding

Self-supervised learning (SSL) has drawn increasing attention in histopathological image analysis in recent years. Compared to contrastive learning which is troubled with the false negative problem, i.e., semantically similar images are selected as negative samples, masked autoencoders (MAE) build SSL from a generative paradigm which is probably a more appropriate pretraining. In this paper, we introduce MAE to histopathological image understanding, and moreover, verify the effect of visible patches in this task. Specifically, a novel SD-MAE model is proposed to enable a self-distillation augmented MAE. Besides the reconstruction loss on masked image patches, SD-MAE further imposes the self-distillation loss on visible patches to enhance the representational capacity of encoder located in the shallow layers. It generates a more effective feature pre-training and benefits downstream applications. We apply SD-MAE to histopathological image classification, cell segmentation and cell detection. Experiments demonstrate that SD-MAE shows highly competitive performance compared with other SSL methods in these tasks. Code is available at https://github.com/irsLu/SD-MAE/