PySpark-Powered ML Models for Accurate Spam Detection in Messages

The process of text classification for the purpose of spam detection relies on the principles of natural language processing (NLP) with the objective of automating the identification and categorization of messages into either spam or ham classifications. Our research endeavor involves the determination of the optimal machine learning model that yields the highest accuracy in segregating undesired messages or emails. The procedure for text classification was executed through the utilization of PySpark, employing a variety of machine learning algorithms such as Naive Bayes, Support Vector Machine, Decision Trees, and Random Forest. The dataset used for the experiments was sourced from Kaggle, a publicly accessible repository. In order to rectify data imbalance, Synthetic Minority Over-sampling Technique (SMOTE) was employed to ensure an equal representation of both spam and ham instances. The outcome of our study prominently designates the support vector machine (SVM) as the most precise machine learning model, demonstrating an F1 score of 97.93 and an accuracy score of 97.9.