Jack of All Trades, Master of None: PMP-Guided Adaptive Multi-Teacher Distillation with Meta-Learning

To enhance the robustness and accuracy of the small model, existing approaches combine adversarial training with knowledge distillation, introducing a comprehensive single-teacher model to improve the performance of the student model (small model). However, due to the limited knowledge of a teacher model, it appears "knowledge gain saturation" phenomenon. Therefore, we propose a PMP-Guided Adaptive Multi-Teacher Distillation with Meta-Learning. Pontryagin’s Maximum Principle is employed to solve the issue of inconsistent teaching objectives among teachers causing distinct optimization directions. Meanwhile, Meta-learning-network is designed to tackle the problem of a student struggling to balance the learned knowledge. A series of experiments conducted on public datasets demonstrate that our approach outperforms the state-of-the-art methods against various adversarial attacks.