Holistic Weighted Distillation for Semantic Segmentation

Channel-wise distillation for semantic segmentation has proven to be a more effective method than spatial-based distillation. By removing the redundant information from the teacher model, the student can focus on specific channel-related pixels, which can be viewed as a weighting of the pixels. However, the standard channel-wise distillation ignores the fact that such importance difference also exists among channels. In this paper, we propose a novel method called Holistic Weighted Distillation (HWD) to address this issue. We calculate the channel divergences between the teacher and the student, and convert them into distillation weights, making the student focus more on learning channels that are not well mastered, thus improving the final model performance. Besides, our method does not introduce additional network structure or back-propagation process, which improves the training efficiency. Experiments on ADE20K, Cityscapes, and COCO-Stuff demonstrate the superiority of our method. The code is available at https://github.com/zju-SWJ/HWD.

I. Introduction

Semantic segmentation is an important and challenging task in computer vision [1]. Due to the huge amount of computation, how to simultaneously achieve high accuracy and efficiency has become a pressing problem for the deployment on mobile and edge devices. Some works attempt to design lightweight networks [2], [3], while others employ knowledge distillation [4] to achieve model compression [5] – [8].