Joint Framework of Curriculum Learning and Knowledge Distillation for Noise-Robust and Small-Footprint Keyword Spotting

Spoken keyword spotting, which is characterized by simplicity and low latency, has been widely used in consumer electronics to facilitate always-on voice interfaces. Small-footprint keyword spotting based on tiny convolutional neural networks can be implemented on resource-constrained, yet energy-efficient, microcontrollers in real time. However, it is difficult for tiny neural networks to learn the noise-robustness properties essential for successful voice interfaces. To overcome this problem, this study proposes a joint framework of curriculum learning and knowledge distillation for noise-robust small-footprint keyword spotting. The proposed joint framework applies noise-mixture curriculum learning to a network that is sufficiently large, to learn various noise situations. Subsequently, knowledge distillation is applied to compress the large network into a sufficiently small network for use in an onboard microcontroller. To enhance the effectiveness of the joint framework, a curriculum learning approach is proposed with a new noise mixture strategy along with knowledge distillation that employs an effective ensemble of neural network snapshots for each curriculum stage. The proposed methods enable large networks to effectively learn noisy situations, thereby transferring noise robustness to small networks. The effectiveness of the joint framework was illustrated on the Google Speech Commands dataset with noise mixtures incorporated from various public noise datasets. The performance of the joint framework was superior in noisy situations compared to that of state-of-the-art noise-robust keyword-spotting methods. Consequently, the proposed framework significantly improves the usability of voice interfaces in consumer electronics.