FSMS: An Enhanced Polynomial Sampling Fusion Method for Audio-Visual Speech Recognition

In recent years, end-to-end audio-visual speech recognition (AVSR) has been proposed with great success. However, due to the heterogeneity between different modal data, an effective audio-visual fusion strategy is in need to learn the correlation between the audio and the video information. In this paper, we propose a fusion strategy in which the high-level features output from audio and video encoders are fed into a siamese neural network with shared weights, and then feature-level masking and polynomial sampling are incorporated to train the fusion of the high-dimensional features of audio and video, achieving inter-modality information alignment. In comparison with the traditional fusion methods as concatenation, it results in improvement of performance. We experimentally demonstrate the effectiveness of this approach on the word-level AVSR task.