The Tail at Amazon Web Services Scale

While tail latency events are individually rare, they are compounded in large-scale distributed systems, leading to poor end-to-end experiences. Networking is one of the key contributors to tail latency, due to network congestion, hardware failures, or transient updates causing high queuing delays and packet drops with millisecond-to-second recovery times. At Amazon Web Services (AWS), we originally designed the scalable reliable datagram (SRD) protocol to decrease these latencies in high-performance computing workloads and distributed machine learning training. SRD utilizes multiple network paths and, combined with microsecond-scale retransmission and path failover, allows the use of the lowest latency paths at the cost of some redundant retransmissions. Additionally, AWS designed its own Flash-based storage media to reduce its tail latency, making network latency the main contributor to tail latency of networked storage applications. SRD was then used as the transport protocol for Elastic Block Storage (the AWS-native storage area network), where high-percentile latency is reduced by as much as 90%.