LORE: Learning-Based Resource Recommendation for Big Data Queries

With the development of modern cloud platforms, an increasing number of users are migrating their data analysis tasks to the cloud. Cloud platforms offer a “pay-as-you-go” model, prompting users to focus on both performance and resource costs. Existing query optimization methods primarily address query performance while neglecting resource costs. Mapping queries to their resource consumption is a complex task. To tackle this challenge, we propose a novel learning-based query resource recommendation method called LORE. LORE efficiently and accurately estimates the optimal resources for queries by leveraging dual information from SQL query statements and query execution plans. We model SQL queries and execution plans as directed acyclic graphs and utilize graph neural networks to derive comprehensive representations. To capture the dependencies among all nodes involved in data transmission within an execution plan, we assign path weights to the dependency edges of each node. Our approach integrates data distribution information and captures both direct and indirect dependencies among plan nodes while avoiding unnecessary redundant computations. Experimental results demonstrate that, compared to traditional and other learning-based methods, the LORE model achieves higher accuracy in predicting the optimal resources for queries.