Connecting MapReduce Computations to Realistic Machine Models

This paper explains how the popular, highly abstract MapReduce model of parallel computation (MRC/MPC) can be rooted in reality by showing how to execute MapReduce computations robustly and efficiently on realistic distributed-memory parallel machines. First, a refined model MRC+ is introduced that includes parameters for total work w, bottleneck work ŵ, data volume m, and maximum object sizes m̂. Then matching upper and lower bounds are established for executing a MapReduce calculation on distributed-memory machines - Θ(w/p + ŵ + logp) work and Θ(m/p + m̂ + logp) bottleneck communication volume using p processing elements. The theorem is formulated in such a way that multiple MapReduce steps can be chained. The result is obtained using a careful combination of several load balancing algorithms some of which may be of independent interest.