A PySpark Based Scalable Model for Churn Prediction using Embedding Models

This study addresses the significant challenge of customer churn, the challenge of customers stopping their association with a service or product through the integration of advanced Natural Language Processing, PySpark, and machine learning techniques. Our dataset comprises a combination of structured and unstructured data, necessitating preprocessing of the data to enhance the predictive accuracy of churn. We employ TF-IDF for feature extraction and implement two distinct word embeddings methods, Word2Vec and SBERT. Leveraging PySpark’s distributed computing capabilities, we efficiently handle large datasets ensuring scalability. Utilizing five machine learning models which include Logistic Regression, Decision Tree, Random Forest, SVM, and Gradient Boosted Trees, we observe that Decision Tree and Gradient Boosted Tree consistently deliver promising results. Notably, the combination of Gradient Boosted Tree and SBERT emerges as the top-performing model across all methods, with accuracy and Fl-score at 0.94. This comprehensive approach, integrating preprocessing, advanced NLP, and PySpark’s scalability, offers businesses a powerful tool to analyze customer interactions, predict potential churn, and ultimately improve customer retention and overall performance.