The Survey of Model Scale Impact on Knowledge Distillation Efficiency in LLM

Large language models (LLMs) have exhibited remarkable capabilities in various natural language processing (NLP) tasks, yet their substantial memory and computational requirements pose significant challenges. Knowledge distillation (KD) has been applied to LLMs to reduce their substantial demands for memory and computation resources. This study aims to find the most efficient most efficient KD method for various scales of LLMs. This study observes the recent advances in KD methods by analyzing eleven state-of-the-art KD methods. We categorize these techniques into five groups: divergence-based, reward-based, data-based, task-specific, and hybrid methods. Each method is evaluated based on its efficacy and efficiency in distilling knowledge into smaller, more deployable models. Our findings reveal two key insights: (1) The optimal KD method is contingent upon various factors, including the LLM's scale, task characteristics, and the desired efficiency, with each approach presenting distinct trade-offs. (2) This study has evolved from simple output distribution “imitation” to sophisticated “teaching” paradigms, where context-aware mechanisms optimize knowledge transfer. This survey provides a foundation for future research by systematically outlining current approaches and their trade-offs. Furthermore, the availability of open source implementations for these KD techniques is paramount.