Audio-Radar SMC-PHD Filtering for Indoor Multi-Speaker Tracking

High-resolution millimetre-wave (mmWave) radar sensors have become increasingly popular in consumer markets. This study addresses the challenge of tracking multiple active speakers in indoor environments using high-resolution radar and microphone array. Through our experiments, we have observed that the Sequential Monte Carlo Probability Hypothesis Density (SMC-PHD) filter, when given point cloud data from a high-resolution radar as input, can provide promising tracking performance. In this work, we add another modality, i.e., audio, to the radar SMC-PHD filtering framework for the active speaker tracking task. Specifically, we use the audio Direction of Arrival (DoA) to guide the particle birth and relocation process in the SMC-PHD filtering framework. Furthermore, we propose a likelihood function that jointly considers the spatial and angular estimation from radar and audio. Experimental results on the RAV4D dataset demonstrate that our audio-radar SMC-PHD filtering approach produces reliable trajectories, especially in the challenging cases such as varying numbers of speakers.