Enhancing Overlapped Speech Detection and Speaker Counting With Spatially-Infused Spectro-Temporal Conformer

Overlapped speech is widely present in conversations and can cause significant performance degradation in speech processing such as diarization, enhancement, and recognition. Overlapped speech detection (OSD) and speaker counting, in particular when the speakers are in the far field, pose challenges as the overlapped part is usually short, and heavy reverberation and noise may present in the conversation scenario. Existing solutions predominantly rely on spectral features extracted from a single microphone signal to perform the detection. In this paper, we propose an innovative approach that is able to efficiently extract spatial features via spatial dictionary learning (SDL) and fuse spatial and spectral features derived from multi-channel array signals. In essence, a spectro-temporal module is employed to extract local information from the spectrogram which is able to better detect overlapping segments under real scenarios. Subsequently, we study two kinds of spatial features: directional statistics, which are projected onto spherical location grids using SDL, and generalized cross-correlation with phase transform (GCC-PHAT) to model the spatial characteristics of speakers. These spatial features are then integrated with the spectral features for the OSD and speaker counting. In addition, a non-parameter magnitude-weighted fusion method is employed to combine the magnitude spectrogram with SDL results, enabling the model to prioritize critical frequency bins. Finally, the performance of the proposed approach is studied under AMI, Alimeeting, and CHiME-6 corpora. Experimental results show that our proposed approach achieves better performance than methods using spectral features only. Furthermore, we also conduct experiments of speaker diarization with OSD to evaluate the detection performance.