Performance and Reliability Effects of Multi-tier Bidding on MapReduce in Auction-Based Clouds

Hadoop has become a central big data processing framework in today's cloud environments. Ensuring the good performance and cost effectiveness of Hadoop is crucial for the numerous applications that rely on it. In this paper we analyze Hadoop's performance in a multi-tier market-oriented cloud infrastructure known as Spot Instances. Amazon Spot Instances (SIs) are designed to deliver a cheap but transient alternative to fixed cost On-Demand (ODIs) instances. Recently, AWS introduced SIs in their managed Elastic Map Reduce offering. This managed framework lets the users design a multi-tier Hadoop architecture using fine grained controls to define the instance types both in terms of capacity, i.e. compute/storage/network, but also in terms of costs, i.e. ODI vs SI. The performance effects of such fine grained configurations are not yet well understood. First, we analyze a set of cluster configurations that can lead to important performance effects that can affect both the running time and the cost of such cloud Hadoop clusters. Second, we examine Hadoop's fault tolerance mechanisms and show the inadequacy of these mechanisms for multi-tier bidding architectures. Third, we discuss directions for making the Hadoop framework more market-aware without losing its focus on extreme scalability.