mmFusion: mmWave-Assisted Mono Speech Enhancement for Multisource Aliasing and Addressing

Separately extracting multiple overlapping target speech signals is a challenge. Microphone arrays can perform areal sound capture but require a large space to achieve good resolution. The emerging millimeter-wave (mmWave) vibration-sensing technique can reconstruct speech by measuring the vibrations of target sound sources. However, because of the weak vibrations of the high-frequency speech component, the key semantic content is likely to be lost. We aim to design a compact speech-sensing method termed as mmFusion to achieve antialiasing high-quality speech perception, which integrates an mmWave radar and a mono microphone. In mmFusion, the antialiasing property of mmWave radio and the high fidelity of the microphone are retained. Also, mmFusion does not rely on any voiceprint information of the speaker as a prior. By generating a fused signal using the microphone signal and the vibration signal of the target, and then enhancing it through a diffusion model, mmFusion can achieve selective high-quality perception of multiple speech sources. The experimental results demonstrate that mmFusion performs well across ten different metrics. In scenarios with overlapping sources and strong noise, mmFusion can independently locate and enhance speech signals from different sources, thereby achieving high-quality speech perception.