A High Accuracy and Ultra-Low Energy Environmental Sound Recognition Processor with Progressive Spectrogram Processing and Adaptive Weight Clustering Based Online Learning

In recent years, environmental sound recognition (ESR) technology has been proposed to detect sounds of interest in the environment [1] [2], which is useful in many applications such as smart homes (e.g. detection of baby crying / door knock), hearing impaired aiding (e.g. detection of car horn / clock alarm) and community security (e.g. detection of glass breaking / gun shot), as shown in Fig. 1. However, the existing works have two main issues. Firstly, the existing ESR algorithms are mainly implemented on GPUs or FPGAs, which cannot meet the energy constraints of AIoT devices, especially when SOTA ESR algorithms heavily rely on deep neural network (NN) to improve accuracy resulting in large computations. Secondly, most of the existing ESR algorithms are trained using public datasets that are recorded in different sound environmental conditions (e.g. microphone, room sound environment) from that of their actual use, leading to significant accuracy degradation. Therefore, in this work, we have proposed an ESR processor which to the best of our knowledge is the first fabricated ASIC-based ESR processor. It achieves high accuracy and ultra-low energy consumption with robustness against sound environmental condition change through several architecture-level techniques, including: 1) Progressive spectrogram processing (PSP) architecture to reduce energy consumption, 2) Split-and-merge processing (SMP) architecture to further reduce the energy consumption, and 3) Adaptive weight clustering based online learning with offline domain-adversarial training (AWCOL) to improve the accuracy against the sound environmental condition change.