Multimodal Speech Recognition Assisted by Slide Information in Classroom Scenes

Multimodal automatic speech recognition (ASR) technology has attracted much attention because it improves the accuracy of speech recognition by adding other modal information. However, most of the existing applications are in English and conference scenarios, and lack of Chinese application scenario task corpus. Online course learning has become a mainstream learning method that requires a lot of speech recognition and transcription. Presentation slides in instructional videos often contain a lot of relevant information and can be an important adjunct to speech recognition tasks. In order to make up for the shortcomings of existing multimodal speech recognition technology in Chinese scenes, this paper focuses on the teaching scenes of Chinese teachers and constructs an audio-visual corpus in the classroom of Chinese teachers. By introducing the technical term text in the slide show as additional modal input, the recognition of homophones and proprietary terms is improved. The corpus consists of 424 teacher videos covering primary and junior high school teaching videos. Based on the corpus, this paper constructs an ASR benchmark pipeline for teacher classroom based on mainstream Paraformer architecture, and uses text information in slides as additional modal input to implement speech recognition tasks. Experimental results show that the accuracy of speech recognition can be improved significantly by using slide text information. The feasibility and effectiveness of this method provide an effective solution for intelligent education field.