Developing an Interview Recording System with Speaker Recognition and Emotion Classification

Dialogue records play a crucial role in meetings and counseling, and the speaker’s information is also necessary. Furthermore, if the speaker’s emotions can be captured, the record can more faithfully reflect the speaker’s reactions at that time. This study aims to develop an interview recording system that integrates speaker recognition and speech emotion classification, providing speech transcription, speaker information, and sentence-level emotion recognition. The prototype system consists of three subsystems: the Meeting Recording System serves as the primary user interface for meeting recording and data statistics; the Data Labeling System is used to correct meeting records and as a tool for data collection; the Voiceprint Management System provides functions for speaker registration and voiceprint management. To train the multi-emotion classification model, we re-labeled 11 hours of audio from the NNIME corpus. After performance evaluation, the F1-score of multi-label emotion classification can reach 0.5115, and speaker recognition accuracy can reach 96.39%, while the text records are generated using Microsoft Speech-To-Text API.