VoiceSeeker: Energy-Efficient and Accurate Audio–Visual Target Speaker Separation on Mobile Devices

We present ${{\sf VoiceSeeker}}$VoiceSeeker, a mobile system that utilizes vision as complementary modality to separate the target speaker’s speech from the input mixture audio. Despite recent advances in deep neural network (DNN)-based audio–visual target speaker separation, important system challenges remain unsolved: separation accuracy drop due to visual feature quality fluctuation, and high resource demand of composite-DNN execution on audio–visual inputs. We design an adaptive pipeline that flexibly switches between audio–visual, audio-only, and no separation depending on the audio–visual input content. For agile adaptation on audio–visual inputs whose content changes rapidly with low temporal correlation, we design a suite of lightweight probing techniques to analyze them on the fly and select the adequate separation path. We also develop a continuous audio-only separation pipeline capable of robustly and efficiently separating the target speaker. ${{\sf VoiceSeeker}}$VoiceSeeker achieves up to 49% energy consumption reduction, 2.2 dB separation accuracy, and 41% latency gain over baselines.