Distil-DCCRN: A Small-Footprint DCCRN Leveraging Feature-Based Knowledge Distillation in Speech Enhancement

The deep complex convolution recurrent network (DCCRN) achieves excellent speech enhancement performance by utilizing the audio spectrum's complex features. However, it has a large number of model parameters. We propose a smaller model, Distil-DCCRN, which has only 30% of the parameters compared to the DCCRN. To ensure that the performance of Distil-DCCRN matches that of the DCCRN, we employ the knowledge distillation (KD) method to use a larger teacher model to help train a smaller student model. We design a knowledge distillation (KD) method, integrating attention transfer and kullback-Leibler divergence (AT-KL) to train the student model Distil-DCCRN. Additionally, we use a model with better performance and a more complicated structure, Uformer, as the teacher model. Unlike previous KD approaches that mainly focus on model outputs, our method also leverages the intermediate features from the models' middle layers, facilitating rich knowledge transfer across different structured models despite variations in layer configurations and discrepancies in the channel and time dimensions of intermediate features. Employing our AT-KL approach, Distil-DCCRN outperforms DCCRN as well as several other competitive models in both PESQ and SI-SNR metrics on the DNS test set and achieves comparable results to DCCRN in DNSMOS.

I. Introduction

Speech enhancement (SE) aims to remove the interference and preserve the speech from the noisy mixture. With the advent of data-driven deep learning approaches, SE models can directly learn the relationship between speech and noise from large paired clean-noisy speech datasets and gain a strong ability in noise suppression. More recent approaches prefer using time-frequency (TF) domain models [1], [2], [3], which have achieved excellent performance by enhancing the noisy speech's real and imaginary components. DCCRN [1] is a popular model for TF domain speech enhancement, and many subsequent works focus on refining this model [4], [5], [6], [7], [8], [9], [10], [11], [12].