CpG Island Detection Using Modified Transformer Model with Pre–trained Embedding

Accurate detection of CpG islands plays a pivotal role in uncovering potential promoter regions for housekeeping and tissue-specific genes, shedding light on critical epigenetic factors contributing to cancer etiology. If formulated as a Named Entity Recognition(NER) problem, different kinds of deep learning models in Natural Language Processing(NLP), specifically transformer-based architectures can be used for CpG island detection which can outperform the traditional rule-based and statistical methods. Pre-training on a generic task with a larger data set and fine-tuning on a domain-specific task with a smaller data set is a well-established approach for many deep learning-based solutions. In this paper, a pre-training and fine-tuning approach is presented for the first time to detect CpG islands. Experiments on the publicly accessible EMBL human DNA database reveal significant improvement over only task-specific training achieving more than 97% accuracy and 83 % F1-score which are better than any existing literature on the same task. The same approach and pre-trained embedding can be utilized for other tasks related to genetic marker identification.