Syntactic-Aware Text Classification Method Embedding the Weight Vectors of Feature Words

Text classification is the process of categorizing sentences into predefined classes on the basis of their meanings or functions. In this context, feature words play a crucial role in representing the core meaning and key information of sentences, reflecting the main content and deep semantics of scientific literature. Optimizing text classification models using feature words is an important task in the information mining of scientific literature. To address text classification considering feature words, this study constructs a set of feature words from a linguistic perspective and quantitatively expresses the weight differences of various feature words on the basis of syntactic structure features. By embedding the weight information, absolute position, relative position, and contextual information of feature words into pre-trained models, the universal text classification models considering feature word vector weights is established. The analysis of hyperparameter space relationships reveals that the hyperparameter settings are not simple linear relationships and that their interactions are often nonlinear and complex. Employing the Bayesian optimization algorithm facilitates the model in finding the optimal hyperparameters while reducing computational costs. Based on the public dataset RCMR 280K, the experimental results demonstrate that considering feature words enables the model to capture sentence structure and features in text classification task, improving the recognition accuracy of the model. Compared with base models (BERT, SciBERT, RoBERTa and ModernBERT) and their regularized models which consider feature words, the improved ModernBERT, which is based on weighted feature word vectors, achieves an average F1 of 95.8% in the recognition tasks of research objective, method, result and conclusion sentences, an improvement of 1.2%, 0.8%, 1.1%, and 0.9%, respectively. In addition, the results show that large language models (LLMs) exhibit robust performance in text classification tasks, with the fine-tuned GLM4-9B achieving superior performance by reaching an average F1 of 91.9%. This model can be applied to sentence recognition and classification tasks in various domains. By updating or replacing the feature word table, the model can be adapted to recognize and classify different types of sentences, providing a theoretical basis or model foundation for other downstream tasks. More details and model weights are public at https://huggingface.co/wmsr22/Classification/tree/main