Traditional and parameter-efficient fine-tuning of LLMs for sentiment analysis in the English and Serbian language

Fine-tuning of large language models is often demanding in terms of computational resources and memory. Consequently, there is a need to explore new methods that can effectively fine-tune these models without compromising performance. In this paper, we compared traditional and parameter-efficient fine-tuning of large language model XLM-RoBERTa, across both high-resourced languages like English and low-resourced languages such as Serbian. The model is fine-tuned for sentiment analysis when it is conducted using text classification. The experiments have shown that parameter-efficient fine-tuning can produce results that are on par with traditional fine-tuning, both for the English and the Serbian language. Furthermore, parameter efficient fine-tuning approach significantly reduces the computational time and memory requirements associated with model training, thereby offering a promising avenue for resource-efficient large language models fine-tuning. In experiments with the English language, we achieved accuracy using the PEFT approach that was only 3% lower than the accuracy obtained through the traditional approach, while reducing training time by 35% and memory requirements by 42%. In experiments involving the Serbian language, we achieved accuracy using the PEFT approach that was only 1% lower than the accuracy obtained through the traditional approach, while reducing training time by 42% and memory requirements by 69%.