Enhancing Airline Operations by Flight Delay Prediction - A PySpark Framework Approach

Airline delay prediction plays a crucial role in the aviation industry, enabling airlines to optimize operations and improve passenger satisfaction. In this research, we propose a comprehensive framework for airline delay prediction using PySpark, a distributed data processing framework. Our study incorporates four widely used machine learning algorithms: Logistic Regression Classifier, Random Forest Classifier, Gradient Boosted Tree Classifier and Decision Tree Classifier. Our experimental results demonstrate the potential of the proposed framework. The identified classifiers when implemented achieve promising results in terms of predictive accuracy, highlighting their suitability for airline delay prediction. By comparing and contrasting these algorithms, we provide insights into their respective strengths and limitations. This research presents a robust framework for airline delay prediction, leveraging PySpark's distributed data processing capabilities. The experimental evaluation of multiple machine learning algorithms highlights their efficacy in accurately predicting flight delays. The deployed prediction system showcases the practical application of our framework, paving the way for improved operational efficiency and enhanced passenger experiences in the aviation industry. From the experiments, logistic regression classifier and random forest classifier had better results.