Enhancing Fine-Tuning of Pre-trained Language Models for Sentiment Analysis Using Metaheuristic Algorithms

Fine-tuning pre-trained language models for specific natural language processing tasks often leads to suboptimal performance due to the vast parameter space and the challenge of finding the most effective model configurations. Traditional fine-tuning methods can be computationally expensive and may not fully exploit the potential of these models, particularly for tasks like sentiment analysis. This work explores the application of metaheuristic techniques, focusing on sentiment analysis using the Stanford Sentiment Treebank 2 (SST2) dataset.We utilize two approaches: Genetic Algorithms (GA) and the Whale Optimization Algorithm (WOA). These methods aim to efficiently search the parameter space and optimize the fine-tuning process of pretrained language models. To enhance efficiency, we incorporate layer freezing techniques along with different crossover strategies. In the results obtained, GA with Simulated Binary Crossover (SBX) and 30% layer freezing achieved 94.02% accuracy, while WOA with SBX crossover and 30% layer freezing achieved 94.67% accuracy. These outcomes highlight the potential of integrating evolutionary algorithms, nature-inspired optimization techniques, and strategic layer freezing with deep learning finetuning processes for NLP tasks.