Using Deep Learning to Predict and Optimize Hadoop Data Analytic Service in a Cloud Platform

Hadoop is a popular computing framework to deliver timely and cost-effective data processing on a large cluster of commodity machines. It relieves the burden of the programmers dealing with distributed programming, and an ecosystem of Big Data solutions have developed around it. However, Hadoop job execution time can be greatly depending on the its runtime configurations and resource selections. Hence, optimizing Hadoop execution still requires a substantial amount of expertise and experiences. To address this challenge, this paper aims to develop a learning-based technique to predict Hadoop job time based on historical execution data, and built a system to optimize its performance in a shared resource cloud environment. While deep learning has shown successes in many application domains, little attention has been paid to apply such technique in job time prediction. In this work, we conducted extensive experimental studies to compare our deep learning prediction method with three other state-of-art regression-based prediction methods in a cloud platform built by OpenStack. The results showed that our prediction method out-performed traditional approaches in most test cases, and improved job performance and cost significantly.