Two-stage fine-tuning for Low-resource English-based Creole with Pre-Trained LLMs

Sentiment analysis adopts natural language processing (NLP) techniques to determine the emotional tone of the text. Sentiment analysis research has predominantly been conducted on commonly spoken languages, such as English and Mandarin Chinese. However, less widely spoken languages, such as Englishbased Creoles, have received limited attention due to the need for large-scale labeled datasets. English-based Creoles are derived from English, resulting in many similarities with English. However, English-based Creoles differ from English in some aspects, making the Creole unintelligible to other English speakers. In sentiment analysis research conducted on Singapore English-based Creole (Singlish), most existing research does not leverage the capabilities of large-language models, such as pretrained BERT. To our knowledge, only Gotera et al.'s work relates to a large-language model; however, their contribution is creating a new pre-trained model for Singlish. However, creating a new pre-trained model is computationally expensive, and pre-training a less widely spoken language will result in ineffective because of the small dataset. Hence, we propose a new two-stage fine-tuning framework for pre-trained models, targeting a low-resource English-based Creole, Singlish. Our proposed framework initially clusters the dataset into two based on each data point's percentage of English words. Then, two-stage fine-tuning is performed by transferring the pre-trained BERT model onto the clustered dataset with the higher English percentage in the first stage. The model is further transferred onto the clustered dataset with the lower English percentage in the second stage. Our proposed framework outperforms the traditional fine-tuning framework, achieving a weighted F1 score of 0.8344 for sentiment analysis.