Enhancing Robustness in Audio Visual Speech Recognition: A preprocessing approach with Transformer and CTC Loss

Audio Visual Speech Recognition (AVSR) is a promising technology for speech recognition that is more robust to noise and other challenging conditions than traditional Audio Speech Recognition (ASR). AVSR systems work by fusing audio and visual features using machine learning algorithms. One of the key challenges in AVSR is developing effective fusion methods. AVSR systems are currently being used in many different applications, having made great strides in recent years despite these obstacles. In this paper, we introduce a preprocessing technique applied on a Transformer model with Connectionist Temporal Classification (CTC) loss that increases the robustness of the AVSR model against noise. We preprocess the training audio clips by adding noise to it, resulting in significant reductions in word error rates and a more efficient training process. The project significantly advances in the field of audio-visual speech recognition and highlights the potential for practical applications, particularly in noisy and challenging environments.