The Declining Role of Lemmatization and Stemming in Modern NLP Pipelines

With the increasing availability of raw text data, Natural Language Processing (NLP) application like predictive modeling has become more prevalent. Consequently, the effectiveness of text preprocessing, particularly text normalization, has become a topic of increasing interest. This study examines the impact of two commonly used text normalization techniques-lemmatization and stemming-and their relevance in the modern NLP pipelines, given the increasing adoption of contextual word embeddings such as Bidirectional Encoder Representations from Transformers (BERT). To achieve these objectives, text data extracted from five diverse datasets were preprocessed using various configurations, and multiple classical and deep learning models were trained on these data using different feature representation techniques, ranging from traditional approaches including Bag-of-Words (BoW) and Term Frequency-Inverse Document Frequency (TF-IDF) to the advanced word embeddings such as Global Vectors for Word Representation (GloVe) and BERT for vectorization. Results indicate that models trained on lemmatized or stemmed text generally achieve lower accuracy compared to those trained on raw or minimally processed text, although results vary by dataset and configuration. However, when using BERT embeddings, models trained on raw text consistently outperform those trained on preprocessed input, achieving up to 10.72% higher accuracy. Additionally, models incorporating lemmatization consistently outperform those using stemming, with accuracy improvements of up to 2.38% when combined with other preprocessing steps, and up to 3.94% when used alone. These findings underscore the importance of applying lemmatization and stemming selectively, rather than as a default preprocessing step because unnecessary use of them could potentially degrade model performance.