BanglaGPT: A Generative Pretrained Transformer-Based Model for Bangla Language

Natural Language Processing (NLP) has entered a new era with the advent of pre-trained language models, paving the way for constructing robust language models. Pretrained transformer-based models such as GPT-2 have become prevalent due to their cutting-edge efficiency. However, these approaches rely heavily on resource-intensive languages, forcing other languages to adopt multilingual frameworks (mGPT). The mGPT model could perform better for low-resource languages such as Bangla because the model has been trained on a diverse dataset spanning multiple languages. Recent studies show that the language-specific GPT model outperforms the multilingual mGPT model. In this research, we have proposed a pretrained monolingual GPT model called BanglaGPT using the objective of causal language modeling (CLM). Due to the lack of available large datasets for NLP tasks in Bangla, we have created a Bangla language model dataset called BanglaCLM using a 26.24 GB Bangla corpus scraped from several public websites. We have used a subword-based tokenization algorithm named Byte-Pair Encoding (BPE) for Bangla and finally trained the Bangla-GPT2 model from scratch using the BanglaCLM dataset. Our pretrained BanglaGPT provides state-of-the-art performance for Bangla text generation with a perplexity score of 2.86 and a loss score of 0.45 on the test set.