A multi-granularity information-enhanced pre-training method for predicting the coding potential of sORFs in plant lncRNAs

Small open reading frames (sORFs) are nucleotide sequences that may be translated into small peptides. Recently, increasing studies have demonstrated that peptides encoded by sORFs in plant long noncoding RNAs (lncRNAs) play a vital role in growth regulation and disease treatment. To accelerate the discovery of lncRNA-encoded peptides, it is essential to predict translatable sORFs in lncRNAs (lncRNA-sORFs) by computational methods. As only a few translatable plant lncRNA-sORFs have been discovered to date, there is a lack of effective methods for characterizing the coding potential of lncRNA-sORFs in data-scarce scenarios. Therefore, a novel method for plant lncRNA-sORFs coding potential prediction using the pre-trained bidirectional encoder representations from transformer (LSCPP-BERT) is proposed. Firstly, the BERT model is trained to extract multi-granularity context information from large-scale unlabeled lncRNA-sORFs through two pre-training tasks. Then, the pre-trained model can be fine-tuned with two additional linear layers for classification. The LSCPP-BERT is featured by a self-supervised pre-training scheme and multi-granularity context information, aiming to enhance the representational power of the network. In addition, an extra pre-training task called contextual relation of lncRNA-sORFs prediction (CRSP) is presented to extract sentence-level information. Experiment results show that the accuracy of LSCPP-BERT is increased by 8.14% compared with state-of-the-art methods. We hope that the proposed method can serve as a reliable tool for the prediction of coding lncRNA-sORFs, thereby further contributing to drug development and agronomical applications.