Distilling Hubert with LSTMs via Decoupled Knowledge Distillation

Much research effort is being applied to the task of compressing the knowledge of self-supervised models, which are powerful, yet large and memory consuming. Existing works generally distill internal features of self-supervised Transformer models. In this work, aiming at more flexibility in the design of the student model, we apply the method of Knowledge Distillation and its more recently proposed extension, Decoupled Knowledge Distillation, to the task of distilling HuBERT. We achieve this by leveraging the cluster prediction pre-training task of HuBERT, which provides valuable targets for the distillation objective. We thus propose to exploit the acquired flexibility to distill HuBERT’s Transformer layers into an LSTM-based model that reduces the number of parameters even below DistilHuBERT and at the same time shows improved performance in automatic speech recognition.