ExpREsS: EneRgy Efficient Scheduling of Mixed Stream and Batch Processing Workloads

Nowadays we see the wide adoption of novel distributed processing frameworks such as Apache Spark for handling batch and stream processing big data applications. An important aspect that has not been examined in these systems is their energy consumption during the application execution. Reducing the power consumption of modern datacenters is a necessity as datacenters contribute over 2% of the total US electric usage. One way of addressing this energy issue is by scheduling the applications in an energy-efficient way. However, efficiently scheduling applications can be challenging as we need to consider the trade-off between the datacenter's energy usage and per application performance requirements. In this work we propose, ExpREsS, a scheduler for orchestrating the execution of Spark applications so that it both minimizes the energy consumption and satisfies the applications' performance requirements. Our approach exploits time-series prediction models for capturing the applications' energy usage and execution times, and then applies a novel DVFS technique to minimize the energy consumption. Our detailed experimental evaluation using realistic workloads on our local cluster illustrates the working and benefits of our approach.