LoRA-Fine Tuned LLaMA-3 for Low-Resource Multilingual Machine Translation

This paper describes a multilingual machine translation system that uses Low-Rank Adaptation (LoRA) to finetune Meta’s LLaMA-3 (8B parameters) to translate low- and medium-resource languages. The study focuses on English (EN) to French (FR), Hindi (HI), Luganda (LG), and Spanish (ES) using limited parallel data to support high-quality translation. The system was trained on a low (3 million) number of sentence pairs from JW300 and OSCAR corpora. Training employs an extensive preprocessing pipeline (cleaning, normalisation, and subword tokenisation with LLaMA-3’s large (128k) SentencePiece vocabulary) and utilises LoRA adapters with QLoRA (4bit quantisation) for efficient finetuning. The finetuned model achieves competitive results on the FLORES-200 dev set for EN to FR, EN to HI, and EN to LG, despite finetuning .01 percent of the model parameters, with BLEU scores of 35.2, 18.4, and 12.1, respectively. Additional evaluations are provided using ChrF, TER, and BERTScore metric evaluations, and the results are discussed in consideration of resource availability. It is reported that the training loss curve typically stabilises after 50k training steps, indicating convergence, and the loss trade-offs for quantisation are examined. In addition, several techniques are experimented with to improve translation quality, including data augmentation with back-translation and filtering the corpus before training the model, which resulted in statistically significant gains, especially for the lowest-resource pair (EN - LG). The paper also discusses reflections on lessons learned for parameter-efficient adaptation in multilingual translation and future work towards universal translation models.