Optimizing Query Performance in Big Data Systems Using Machine Learning Algorithms

The retrieval of information from extensive datasets in an efficient manner is critical in the era of big data. Conventional optimization methods encounter difficulties in managing the multitude of intricate and varied queries that big data systems confront. To optimize query efficacy in big data systems, this article introduces an innovative strategy that uses machine learning (ML) algorithms. To forecast optimal query execution strategies, our methodology entails training machine learning models using historical query execution data. The methodology consists of two components: First, characteristics are extracted from query plans, encompassing query patterns and statistics pertaining to the underlying data. Furthermore, to forecast the efficacy of a specific strategy, ML models, including decision trees, neural networks, and gradient-boosting machines, are trained using these characteristics. The findings were persuasive. In particular, for intricate queries, our ML-based optimizer enhanced query performance by as much as 60 % when compared to conventional optimizers. Changing duties and data distributions were also accommodated by our system, demonstrating its robustness and adaptability. Our research establishes a foundation for an emerging class of adaptive query optimizers capable of acquiring knowledge and adjusting to the dynamic characteristics of data and queries. By integrating machine learning capabilities with big data systems, we present a novel solution that significantly enhances the speed and efficiency of data retrieval.