New Method for Intent Recognition Based on Audio and Text Multimodality

This paper introduces an innovative intent recognition approach that leverages the complementary strengths of audio and text multimodalities. Initially, the method captures the user’s raw voice data and transcribes it into textual information. Subsequently, the Roberta text intent recognition model is employed for extracting textual features and discerning intent, where the first cross-entropy is computed. In parallel, the unsupervised pre-training model wav2vec is utilized to extract audio features directly from the original voice information, and intent is recognized, resulting in the second cross-entropy. Furthermore, the method integrates intent recognition by concatenating the extracted text and audio features, yielding the third cross-entropy. The ultimate intent recognition is achieved through a weighted calculation of these three cross-entropies. The weights are derived through training the Roberta text intent recognition model in conjunction with the wav2vec unsupervised pre-training model. This approach, particularly pertinent to the field of Human-Computer Interaction (HCI), minimizes the risk of error propagation from inaccurate textual transcriptions, thereby enhancing the precision of downstream tasks. By advancing the interface between linguistic and acoustic data analysis, this method highlights the transformative impact of integrating advanced natural language processing and audio signal processing technologies in HCI applications. The proposed method underscores the significance of applying sophisticated audio-visual integration techniques in engineering disciplines to achieve robust and accurate intent recognition in modern interactive systems.