Performance Evaluation of Apache Hadoop, Spark, and Flink for Batch Processing of Big Data: A Comparative Analysis

The work explores the demand for Big Data processing and delves into the functioning of large-scale data processing architectures, focusing on batch and real-time processing. The experiment conducted analyzes the execution time of Apache Hadoop (AH), Apache Spark (AS), and Apache Flink (AF) tools. Results indicate that Spark outperformed Flink and Hadoop across all experiments, demonstrating notable speed advantages. In the first experiment with a 1 GB data source, Spark was 186% faster than Flink and 251% faster than Hadoop. Similarly, in the second experiment with a 3 GB data source, Spark surpassed both competitors, being 233% faster than Hadoop and 334% faster than Flink. Processing a 5 GB data source further highlighted Spark's superiority, with a 197% improvement over Hadoop and 316% over Flink. Despite the absence of parallelism and a distributed execution environment, the findings unanimously conclude that Spark achieved superior performance in batch processing large datasets in a pseudo-distributed cluster setting.