Helix: A Cloud-based Big Data Orchestration Tool for Automating Distributed Job Management with Hadoop and Spark

Helix is a Big Data orchestration tool that automates the submission, management, and execution of Hadoop and Spark jobs on cloud-based clusters, focusing on academic users and small-scale research projects. Using Spring Boot, Amazon EMR, and S3, Helix reduces operational overhead by dynamically provisioning or reusing clusters based on job requirements. In performance evaluations, Spark outperformed Hadoop by 30% on datasets larger than 50 GB. Cluster reuse lowered provisioning time from 8–12 minutes to 1.5 minutes, making Helix time-efficient and cost-effective, particularly with A WS free-tier usage. The paper presents Helix's architecture, implementation, evaluation, and comparison with existing tools. Key contributions include: (1) automation of job orchestration and cluster management, (2) integration with A WS for cost-efficiency, (3) performance optimization with Spark, and 4) accessibility for small-scale research. Future work explores multi-cloud support and enhanced scheduling. The paper is structured into Introduction, Related Work, Architecture, Implementation, Evaluation, and Conclusion.