Sparse MoE Students for Efficient Knowledge Distillation

We present a sparse Mixture-of-Experts student architecture for knowledge distillation that replaces conventional dense students with a modular design of lightweight, class-agnostic experts dynamically routed by input-conditioned gating. Our approach provides three key benefits: (i) accuracy–efficiency Pareto superiority over dense students by activating only a subset of experts per input, (ii) loss-agnostic plug-and-play compatibility, consistently improving performance across diverse Knowledge distillation objectives, and (iii) class-agnostic expert decomposition, enabling generalization without label supervision. To ensure fairness, we construct both dense and Mixture-of-Experts students with comparable parameter budgets and systematically evaluate multiple gating strategies, including soft, top-k, and attention-based variants. Experiments on CIFAR-10 and CIFAR-100 under four distillation methods show that attention-gated MoE students consistently outperform dense baselines at equal capacity, achieving up to +1.7% accuracy gains with an average 25% FLOPs reduction. These findings demonstrate the structural advantages of sparse Mixture-of-Experts students as efficient, generalizable, and loss-agnostic alternatives to dense models in knowledge distillation.