Amalgamating Knowledge for Comprehensive Classification with Uncertainty Suppression

Knowledge distillation(KD) aims to obtain a lightweight student network with the target dataset's pre-trained network(s). In practical applications, the student network distilled on one dataset may fail to make fine-grained classifications of multiple categories(such as birds and dogs). To this end and to make better use of various datasets' pre-trained models, knowledge amalgamation (KA) strives to integrate the knowledge of multiple expert models trained on different datasets to attain a student network with multi-expert knowledge. Proposed KA methods for image classification ignore the problem that teacher networks may encounter with untrained class samples and provide misleading guidance to the student network. To address this problem, we propose a knowledge amalgamation framework based on uncertainty suppression. A series of experiments demonstrate the effectiveness of our framework; some of the experiments yield an accuracy improvement of 2% compared to the proposed methods.