Neural Scoring: A Refreshed End-to-End Approach for Speaker Verification in Complex Conditions

Modern speaker verification systems primarily rely on speaker embeddings, followed by verification based on cosine similarity between the embedding vectors of the enrollment and test utterances. While effective, these methods struggle with multi-talker speech due to the unidentifiability of embedding vectors. In this paper, we propose Neural Scoring, a refreshed end-to-end framework that directly estimates verification posterior probabilities without relying on test-side embeddings, making it more robust to complex conditions, e.g., with multiple talkers. To make the training of such an end-to-end model more efficient, we introduce a large-scale trial e2e training strategy, where each test utterance pairs with a set of enrolled speakers, thus enabling processing of large-scale verification trials per batch. Experiments on VoxCeleb dataset demonstrate that Neural Scoring consistently outperforms both the baseline and competitive methods across various conditions, achieving an overall 70.36% reduction in Equal Error Rate compared to the baseline.