LipReading for Low-resource Languages by Language Dynamic LoRA

Lipreading in low-resource languages is a highly demanded yet extremely challenging task. Recent research has shown that transferring the lipreading knowledge from data-rich languages to low-resource ones can improve the performance. However, unique lip movements in low-resource languages and heterogeneous syntactic structures and lexical conventions across languages render such knowledge transfer ineffective. To address these challenges, we introduce a Dynamic Low-Rank Fine-tuning to adaptively adjust model parameters for learning a set of basic lip shapes shared across languages, thereby better recognizing lip movements, especially those unique to low-resource languages. Moreover, we design an instruction tuning on multilingual LLMs to enhance the model’s cross-lingual text mapping capability. Experiments conducted on low-resource language datasets, Italian and Portuguese, demonstrate the effectiveness of our proposed method.