Robust Audio-Visual ASR with Unified Cross-Modal Attention

Audio-visual speech recognition (AVSR) takes advantage of noise-invariant visual information to improve the robustness of automatic speech recognition (ASR) systems. While previous works mainly focused on the clean condition, we believe the visual modality is more effective in noisy environments. The challenges arise from the difficulty of adaptive fusion of audio-visual information and the possible interferences inside the training data. In this paper, we present a new audio-visual speech recognition model with a unified cross-modal attention mechanism. In particular, the auxiliary visual evidence is combined with the acoustic feature along the temporal dimension in the unified space before the deep encoding network. This method provides a flexible cross-modal context and requires no forced alignment such that the model can learn to leverage the audio-visual information in relevant frames. In experiments, the proposed model is demonstrated to be robust to the potential absence of the visual modality or misalignment in audio-visual frames. On the large-scale audio-visual dataset LRS3, our new model further reduces the state-of-the-art WER for clean utterances and significantly improves the performance under noisy conditions.

1. INTRODUCTION

Automatic Speech Recognition (ASR) can be a difficult task when the speech signal gets distorted by noises. As in the conversation between humans, visual modality sometimes becomes necessary to understand speeches accurately since it’s invariant to the presence of noise. Audio-Visual Speech Recognition (AVSR) is the task of generating text transcriptions from both the audio and the auxiliary visual evidence. Although many kinds of visual sources [1] can be useful for speech recognition, lip motion [2] [3] is considered the most related and beneficial evidence in the literature.