Faster Data-access in Large-scale Systems: Network-scale Latency Analysis under General Service-time Distributions

In cloud storage systems with a large number of servers, files (e.g., videos, movies) are typically not stored in single servers. Instead, they are split, replicated (to ensure reliability in case of server malfunction) and stored in different servers. We analyze the mean latency of such a split-and- replicate cloud storage system under general sub-exponential service time distribution, which encapsulates most of the practical heavy-tailed distributions. We present a novel scheduling scheme that utilizes the load-balancing policy of the power of d (≥2) choices. Exploiting the double exponential queue length property of this policy ([1]), we obtain tight upper bounds on mean latency. An alternative to split-and-replicate is to use erasure-codes, and recently, it has been observed that they can reduce latency in data access (see [2] for details). We argue that under high redundancy (integer redundancy factor strictly greater than or equal to 2) regime, the mean latency of a coded system is upper bounded by that of a split-and-replicate system (with same replication factor) and the gap between these two is small. For example, when specialized to an exponential service time distribution, our formulation recovers the result of [3], (which uses erasure codes) upto a constant factor. We also validate this claim numerically under different service distributions such as exponential, shift plus exponential and the heavy-tailed Weibull distribution and compare the mean latency to that of an unsplit-replicated system. We observe that the coded system outperforms the unsplit-replication system by at least 20% for all three distributions and all possible arrival request rates. Also, we analyze the tail latency for data centers memoryless servers. Furthermore, we consider the mean latency for an erasure coded system with low redundancy (fractional redundancy factor between 1 and 2), a scenario which is more pragmatic, given the storage constraints ([4]). However under this regime, we restrict ourselves to the special case of exponential service time distribution and use the randomized load balancing policy namely batch-sampling. We obtain an upper bound on mean delay that depends on the order statistics of the queue lengths, which, we further smooth out via a discrete to continuous approximation.