Systematic Review of Fine-tuning Approaches for Large Language Models in the Context of Sinhala

The rapid development of large language models (LLMs) such as GPT-3, GPT-4, LlaMA, and mBERT has significantly advanced the natural language processing (NLP) field across many widely spoken languages. However, the effectiveness of these LLMs on low-resource languages such as Sinhala remained unexplored. With the scarcity of high-quality data and linguistic resources, fine-tuning LLMs in Sinhala has gained significant challenges. With a focus on Sinhala specifically, this systematic review addresses the methodologies and outcomes of adapting large language models for underrepresented languages. The purpose of this review is to synthesize existing research, identify effective strategies, contributions, and improvements, and highlight the challenges and opportunities in this domain. By analyzing various approaches, including data augmentation, transfer learning, cross-lingual transfer, parameter-efficient fine-tuning, adapter tuning, and model adaptation likewise, this review provides a comprehensive and precise idea of the current state of LLM fine-tuning on low-resource languages like Sinhala. The results highlight the importance of using creative methods and collaboration to improve LLM performance and usefulness in linguistic environments with limited resources.