A Big Data Optimization Comparison using Apache Spark and Apache Airflow

While comparisons between Apache Hadoop and Apache Spark are well-documented, there has been limited research comparing Apache Spark with Apache Airflow, especially in terms of speed and memory usage. With Apache Airflow's recent introduction of dynamic task mapping, which performs similar functions to Apache Spark's map operation, a detailed comparison between the two tools has become increasingly relevant. A comparison in these areas would provide valuable insights for the Big Data Science community, helping determine which methods are better suited for tasks requiring high speed and efficient memory usage. This study focuses on comparing the Apache Spark Map function and Apache Airflow Dynamic Task Mapping function on two key metrics: memory utilization and computation speed. Specifically, we evaluate their performance in sorting formatted electrocardiogram sensory data. We hypoth-esize that Apache Spark will demonstrate faster processing times due to its advanced in-memory processing and sorting algorithms. However, this speed advantage is expected to come with higher memory usage compared to Apache Airflow. Our findings provide actionable insights into the strengths and limitations of these tools, guiding data scientists and engineers in choosing the most suitable framework for specific big data processing tasks. These results are particularly relevant for large-scale data sorting and transformation operations, contributing to informed decision-making in the Big Data Science community.