TinyDNABERT: An Efficient and Optimized BERT Model for DNA-Language in Genome

Transformers are not new in the world of language processing and have been a core component of the ongoing Generative AI trend. Since DNA can be considered to be analogous to natural human language, there have been attempts made to understand the language of DNA using Transformer models, such as DNABERT. However, one problem that remains with these models is that they are computationally very intensive, taking up large resources to perform training and thus the overall inference using the model is slowed down. To address this problem, in this work we present a smaller model viz. TinyDNABERT which is at least 100 times smaller than DNABERT and uses much less resources to train while attaining comparable performance on various downstream tasks such as prediction of promoters, splice sites and Transcription Factor Binding Sites (TFBSs). We further extend this model by using Low-Rank Adaptation (LoRA) which is an efficient fine-tuning technique employing low-rank matrices and lesser number of trainable parameters. As both TinyDNABERT and TinyDNABERT+LoRA are optimised for scenarios which involve less computational resources as compared to DNABERT, they show quite competent performance for fine-tuning tasks while having a much lesser size than DNABERT, thereby providing a balance between performance and size.