Self-Supervised Learning for Online Speaker Diarization

Speaker diarization deals with the issue of “who spoke when” which is tackled through splitting an utterance into homogeneous segments with individual speakers. Traditional methods were implemented in an offline supervised strategy which constrained the usefulness of a practical system. Real-time processing and self-supervised learning are required. This paper deals with speaker diarization by relaxing the needs of reading the whole utterance and collecting the speaker label. The online pipeline components including feature extraction, voice activity detection, speech segmentation and speaker clustering is implemented. Importantly, an efficient end-to-end speech feature extraction is implemented by an unsupervised or self-supervised method, and then combined with online clustering to carry out online speaker diarization. This feature extractor is implemented by merging a bidirectional long short-term memory and a time-delayed neural network to capture the global and local features, respectively. The contrastive learning is introduced to improve initial speaker clusters. The augmentation invariance is imposed to assure model robustness. The online clustering based on autoregressive and fast-match clustering is investigated. The experiments on speaker diarization over NIST Speaker Recognition Evaluation show the merits of the proposed methods.