PN-AMLTrans: A Semantic Classification Method Based on Parallel Network of Transformer and Attention Perception Mechanism

In recent years, pre-trained language models based on Transformer have brought significant breakthroughs to natural language processing (NLP) tasks. Their outstanding performance in general text understanding enables such models to handle a variety of tasks. However, large language models exhibit problems such as the inability to provide a definitive label for topic identification tasks after interactive prompts and difficulties in controlling the label range. In specific application scenarios, the performance of large models cannot accurately perceive the demands of the tasks due to insufficient data. To address this issue, we propose a text semantic classification method called PNAMLTrans. PN-AMLTrans mainly consists of three modules: Non-Adjacent Sequence Learning Module, AMLNet, and AvgSoftmax Classification Module. In model computation, we first use the BERT pre-trained model to learn the initial features of the text, extracting complete semantic representations of the text as much as possible. Next, we utilize the deep residual connection mechanism to serially connect the Transformer Encoder for learning word embedding semantic matrices. Simultaneously, the attention scores from each layer of the Transformer Encoder are input into AMLNet to extract the mapping relationship between attention scores and classification clues. Finally, effective classification with key weight amplification is achieved through joint weighted calculation using average pooling and Softmax. Experimental results show that the model achieves $98.1 \%$ generalization performance on the Chinese topic recognition dataset THUCNews and $83.5 \%$ state-of-the-art performance on the English topic recognition dataset arXiv-10, validating the effectiveness of our method across different languages and demonstrating the effectiveness of the PN-AMLTrans model in specific applications.